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ABSTRACT 

The purposes of this research were to evaluate the 
cost effectiveness of using Asynchronous Computer Conferencing (ACC) 
and to develop guidelines for effectively conducting high quality 
military training using ACC. The evaluation used a portion of the 
Engineer Officer Advanced Course (EOAC) as a test bed. Course 
materials which taught the same content presented in the resident 
course were developed for the remote, asynchronous presentation. This 
delivery system was called the System for Managing Asynchronous 
Remote Training (SMART). The efficacy of remote presentation was 
compared to that of the resident program with regard to completion 
rate, performance, acceptability, and cost. A summative evaluation 
determined the cost effectiveness of the SMART presentation as 
compared with the resident training. A formative evaluation and 
literature review were used to develop guidelines for conducting ACC 
training. Summative findings indicated: (1) ACC training costs less 
than resident training; (2) there were no differences no differences 
between resident and ACC students on objective performance measures; 
(3) ACC students perceived greater learning benefits than resident 
students; (A) ACC training had greater user acceptance; and (5) 
resident training takes less time than ACC training and has a better 
completion rate. The formative findings indicated that: (1) deadlines 
and group activities were the most effective pacing total of 8 hours 
of work on a course per week is a reasonable total of eight hours of 
work on a course per week is a reasonable student requirement; (A) 
immediate feedback is preferable to delayed feedback; and (5) support 
communications, such as a telephone hotline, are critical to the 
success of an ACC course. This report provides guidelines for how ACC 
courses should be implemented to maximize performance and acceptance. 
It also presents evidence supporting the cost effectiveness of ACC as 
a method of remotely delivering training to the Army Reserve 
Components . (DB) 
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FOREWORD 



Limited time and wide geographical dispersion of both units 
and individuals in the National Guard and Army Reserve, i.e., the 
Reserve Component (RC) make it difficult and costly for soldiers to 
travel to branch schools for training. Therefore, the RC is 
exploring alternatives that will use technology to bring training 
and educational opportunities to the soldiers' homes. One of these 
alternatives is the creation of remotely conducted classes in which 
individuals are linked with each other and their instructors using 
asynchronous computer conferencing. 

This report summarizes the findings on the effectiveness and 
costs as well as makes recommendations for course design and 
teaching in the RC using asynchronous computer conferencing. The 
report was developed by the ARI-Boise office within the charter of 
the Training Technology Field Activity-Gowen Field, whose mission 
is to improve Reserve Component Training effectiveness and 
efficiency through the testing and application of technology. The 
research task supporting this mission is entitled "Application of 
Technology to Meet Reserve Component Needs" and is organized under 
"Training for Combat Effectiveness" program area. The National 
Guard Bureau, Forces Command, and TRADOC HQ sponsored this project 
under the Memorandum of Understanding signed 12 June 1985 that 
established the office. Project results have been briefed to 
TRADOC HQ, Forces Command, Office of the Chief, Army Reserve, and 
the National Guard Bureau. 



EDGAR M. JOHNSON 
Technical Director 



DISTRIBUTED TRAINING FOR THE RESERVE COMPONENT: REMOTE DELIVERY 
USING ASYNCHRONOUS COMPUTER CONFERENCING 



EXECUTIVE SUMMARY 



Research Requirements: 

To- (1) evaluate the cost-effectiveness of using Asynchronous 
r^Ztlr conferencing (ACC) techniques to provide high quality 

training* 
Procedure: 

The evaluation used a portion of the Engineer Off icer Advanced 
~ 5 as a test bed. Course materials which taught the 

SS^cffipS^S^ resident --se aevelopea tor 

l%llV^™r^ -y ed "?o that 

??MART> The efficacy of remote presentation was compared to that 
of^he' resident program with regard to throughput, performance, 
acceptability, and cost. 

The current state of RC training and the potential for remote 

s£"Ts d u^^ 

for conducting ACC training. 

Findings: 

The summative findings were: 

1. ACC training was more cost-effective than resident 
training in that: 

a ACC performance (tests, homework, practical 

* exercises) did not differ from resident 
performance. 

b Costs for development and operation of an ACC 

* course were less than operating a comparable 
resident course. 

o romnarisons of the amount students perceived they learned 
2 ' during ?he course showed that ACC students felt they 
learned more than did the resident students. 
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The formative findings were: 



1. The imposition of deadlines was the single most effective 
method of ensuring that students worked through the course 
at a desirable pace. The use of group activities also 
served as a pacing aid. 

2. The availability of group interaction opportunities 
positively impacted students 1 motivation to participate in 
the course . 

3. Students can be expected to spend eight hours per week 
working on a SMART course. This eight hours is inclusive 
of administrative time for activities such as interacting 
with the computer and organizing information. 

4. Students preferred immediate feedback to delayed feedback. 
If feedback must be delayed , the course structure should 
permit students to move on to another assignment while 
awaiting feedback, wherever practical. 

5. The availability of support communications, such as a 
telephone hotline, is critical to the success of a SMART 
course. 

Utilization of Findings: 

This report presents evidence supporting the cost- 
effectiveness of ACC as a method of providing remotely delivered 
training to the RC. It also provides guidelines for how such 
courses should be implemented to maximize throughput, performance 
and acceptance. 



ASYNCHRONOUS COMPUTER CONFERENCING FOR REMOTE DELIVERY OF RESERVE COMPONENT 
TRAINING 
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Introduction 



statpmpnt nf t he Problem 

In its efforts to maintain overall readiness, the U.S. Army is faced 
with some unique challenges in training its two Reserve Components (RC), the 
Army Reserve and the Army National Guard. Training is a key aspect ot 
readiness and, as described below, it is many times more difficult to 
adequately train the RC than it is to train the Active Army. The importance 
of a well trained RC cannot be overemphasized, for it makes up more than 50/ o 
of the total Army strength. 

Background on Tra ining the RC 

Traininq is the responsibility of the unit commander, who may have many 
different military occupational specialties (MOSs) to train. Typically, RC 
commanders have only 39 training days available each year distributed over 
12 weekends and a two week annual training session. In comparison, their 
Active Army counterparts train their units continuously throughout the 
training year. 

RC units are scattered across the United States and overseas at more 
than 4,000 armories and reserve centers. As a result, there are large 
numbers of soldiers with low geographical density. Low density MOSs often 
do not justify local courses so travel to resident/branch schools is 
required. Resident schooling is commonly viewed as the best the Army has to 
offer, but training must accommodate civilian job ana family 
responsibilities. Thus, a trip to the branch school may not be an 
acceptable alternative for many members of the RC. 

Current RC Education Options 

Usually, members of the RC complete course requirements through a 
combination of resident branch schools, such as the Engineer School .Reserve 
Forces (RF) schools, and the Army Correspondence Course Program (ACCP). 

Resident training yields not only a high throughput (number of soldiers 
completing), but also high quality instruction. Eisley and Viner (1989) 
conducted a nationwide survey of RC opinions on training and reported that 
soldiers prefer resident classes to other training options because they are 
perceived as being of higher quality (i.e., as having good instructors, 
course content, and facilities). However, resident courses are not very 
accessible to the RC soldier, who must juggle job and family schedules to 



ttend Gaining attendance at resident school is more difficult than 
gaining attendance at other options (Eisley and Viner, 1989). Further, the 
cost to the Army, in terms of travel, per diem, and pay, is quite high. The 
cost per RC student for seven modules of the Engineer Officer Advanced 
Course (EOAC) is estimated by TRAOOC Headquarters to be $40. 5K. 
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RF School training, on the other hand, sacrifices some quality control 
for greater availability and lower costs. RF school data indicate a per 
student cost of S3.3K for one phase of EOAC. RF courses tend to be easier 
to fit into civilian commitments. Eisley and Viner (1989) reported that 
soldiers favor RF schools with respect to ease of meeting schedules, class 
openings, and funding to support attendance. However, the quality of RF 
courses is not always consistent across locations, since maintaining a pool 
of qualified instructors is difficult for some MOSs (Allen Corporation, 
1986). 

Correspondence courses, although readily available and low in cost ($48 
per student for one phase of EOAC, Allen Corporation, 1986), have much lower 
throughput and quality of training than the other two alternatives. In many 
content areas, ACCP materials may not be doctrinally or technically correct, 
due to long revision cycles (Allen Corporation, 1986). Soldier acceptance 
of correspondence training is lower than acceptance of any other training 
option (Eisley and Viner, 1989). As a result, the dropout rate for 
correspondence courses is quite high, often exceeding 50% (Allen 
Corporation, 1986). 

Purpose 

An ideal training option for the RC would be one that minimizes c6st 
and maximizes quality, throughput, and availability. High quality training 
that could be delivered to the soldier's home or armory/reserve center 
should provide good throughput and acceptance and meet the ideal if it were 
affordable. A potential method of providing such training is remotely 
delivered, computer-mediated training. The practical implications and 
cost-effectiveness of providing such training had been, however, largely 
unexplored. 

The purpose of this project, then, was to investigate an innovative 
alternative to RC training: the System for Managing Asynchronous Remote 
Training (SMART). SMART is a distributed, computer-mediated training system 
capable of delivering training to soldiers' homes and/or armories/reserve 
centers. Of interest are the quality, acceptability, throughput, and cost 
of this system. Specific requirements were to: (1) evaluate the cost- 
effectiveness of using Asynchronous Computer Conferencing (ACC) techniques 
to provide high quality, remotely delivered training to the RC; and (2) 
develop guidelines for conducting such training. 

Overview of SMART 

What is SMART? 

SMART is a computerized distributed training system, the functions of 
which are to provide: 

communication, 
-- a combination of delivery media, and 
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course management 

and the characteristics of which are to support: 

-- geographically distributed training, 

-- asynchronous and synchronous training, and 

-- computer mediated training and communications. 

As is illustrated in Figure 1, the SMART concept provides a 
communication system, allowing an instructor to communicate with students, 
students to communicate with the instructor, and students to communicate 
with each other, all in a distributed fashion. Each student and instructor 
has a personal computer that is networked through a host mainframe computer 
using exiting telephone lines. Other names for this kind of communication 
include asynchronous computer conferencing (ACC) and computer-mediated 
communication (CMC). 

The SMART concept also provides a combination of delivery media All 
types of instruction, including paper-based material s, computer assisted 
instruction (CAI), storyboards (computerized slide presentation), formal and 
informal discussions, problem solving groups, peer tutoring, expert groups 
practical exercises, simulations, video- and audio-tapes, interactive video 
disks (IVD), and hands-on activities, can be accessed via 

Final ly, the SMART concept provides a course management system, 
allowing the instructor to control and administer lessons, exercises, and 
tests, while maintaining rosters, grade books, and attendance re «rds 
SMART also provides mechanisms by which the instructor can provide feedback 
on the performance of various activities to soldiers. 

The distributed nature of SMART means that geographically distant 
learners can participate in the instruction without the need to come 
together in one location. Students and instructional staff can be located 
anywhere there is access to a telephone. Students can work from their 
hSmes? armories, or reserve centers, and instructors can teach from branch 
schools RF schools, or their homes. With portable computers, students and 
instructors can continue their participation in classes even when 
travelling. 

Most of the instruction delivered via SMART is accessed 
asynchronously. That is, not everyone must participate at the same time. 
This flexibility of scheduling makes SMART quite adaptive to persona time 
constraints. However, it also means that there are built-in time delays 
before all students receive the instruction and that certain activities, 
such as group discussions, will take longer than they would in the 
face-to-face environment. When time delays are not practical SMART allows 
. for synchronous communication. Here, all students access SMART at one time 
and work together to accomplish a given task. When the task is completed, 
they return to the asynchronous mode. 
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Computerized means for soldiers end instructors to communicate 
asynchronously from distributed locations. 
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Figure 1. The SMART System 
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Because SMART is computer-mediated, it requires that each student have 
a computer with a modem, to enable computer communication over the 
telephone, and appropriate software to facilitate computer communication and 
learning activities. It also requires telephone links, a host computer 
which supports the uploading (transmitting to someone) and downloading 
deceiving from someone) of information, and a certain amount of learning on 
the Dart of the user. The system facilitates meaningful, connected 
discussions by a group of students. These discussions take place on their 
computers as students observe written input from each other and comment upon 
it This distributed training system also allows instructors and schools to 
take advantage of the fact that every student has a computer, facilitating 
the use of computer-based training. 

The Distributed Training Situation: Students and Instructors 

As in some of the other distributed learning situations, such as 
correspondence, students learning via SMART are experiencing a tabletop 
learning environment -- they are likely to be sitting at the kitchen table 
orTa desk in their homes, working alone in the sense of not having other 
students physically present. Because they are in the RC, they are fitting 
their training around their daily routines and different students may be 
working on very different schedules. These factors combine to make the 
sSART learning y experience quite different from that of a student : taking a 
resident course Figure 2 depicts segments of a "day in the life of a 
re dent studTnt and 9 a SMART student, highlighting differences between the 
two. Appendix A shows what the SMART student sees and does during a SMART 
computer session. 

The fact that SMART is characterized by being distributed asynchronous 
and synchronous, computer-mediated training and communication also impacts 
the SMART instructor with respect to both work schedule and duties 
performed. The emphasis on computer mediation and independent learning 
shifts the emphasis of the role of the instructor from primarily that of a 
deiverer of instruction to more of that of a facilitator and a counsellor. 
That is, the SMART instructor spends less time providing course content vna 
lecturing or other classroom techniques than does the resident instructor 
and more time guiding students through the learning experience by directing 
their studies, answering questions, and providing performance feedback. 

Figure 3 gives highlights of a "week in the life" of a SMART 
instructor. For the purposes of the illustration it is assumed tnat the 
instructor works full time teaching one course. Note that, ? hough the 
instructor works for 40 hours during the week, that time is allocated 
according to student needs rather than as an eight to five workday 
Further, to ensure timely feedback, it is necessary for the instructor to 
check in on the computer on at least a daily basis. 
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Typical Day 


Resident Student J 


MART Student I 


lotes 


1 0700 
0800 

0930 


Jake up & take I 
a shower 

Report to the J 
first class of 
the day 

Take a break & 
chat with other 
students 


Drive to work 1 

< 
( 

Arrive at the 1 
office & begin the < 
work day 

Try to reach a 
classmate by phone; 
no luck 


rhe resident student 
:an devote the entire 
Jay to course work; 
the SMART student 
devotes more than a 
third of the d*y to 
civilian employment 


0945 
1145 

1300 


Begin a smal 1 
group exercise 

Finish the 
group exercise; 
break for lunch 

Complete a CAI 
lesson at the 
computer lab 


Get work assignment 
to write a major 
report by Friday 

Classmate returns 
call ; spends 20 
minutes scheduling 
a synchronous 
meeting for Friday 

Eat lunch at desk 
while doing a SMART 
reading assignment 


The resident student 
has face-to-face 
interaction with 
classmates so group 
exercises are 
accomplished quickly 
in a pre-scheduled 
time period; the 
SMART student must 
coordinate group 
work adding admin 
time to the activity 


1400 


Go to lecture 


Resume office work 




1700 


Return to quarters 


Drive back home 




180 


Have dinnei 
with classmates 


Have dinner with 
family- 




1930 


Do a homework 
assignment 


Check kids' 
homework 




2040 
2330 


Watcn Monday 
night football 

Go to bed 


WOrK on onMi\ i — uu 

a CAI and a paper- 
based exercise 

Go to bed 


Onlv after meetina 

\J III J U 1 w V* 1 til w ^ \+ i « • ^ 

civilian job & 
family requirements 
can the SMART student 
begin his or her 
studies 



Figure 2. A day in the lives of a resident and a SMART student 
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SMART Instructor 



Notes 



Monday 

0800-1100 Go online & receive messages & 
scores on activities, write 
replies & feedback & upload 
(download/upload) 
1100-1500 Update records of soldiers' 
progress; phone students who 
have fallen behind 
1500-1700 Take hotline call from a 
student having software 
problems; send new disks 

Tuesday 

0800-1000 Download/upload 
1000-1100 Obtain clarification from 
proponent school on topic 
about which soldiers have 
questions 



Wednesday 

0800-1100 Download/upload 
Thursday 

0800-1200 Download/upload 
1200-1600 Grade exams & homeworks 

Friday 

0800-1100 Download/upload 
1200-1400 Add funds to computer accounts 
1800 Go online for synchronous 
meeting 

1830-2200 All students are ready to begin 
the exercise; give the 
assignment & watch progress 

2200-2300 Grade group assignment & put 
feedback onl ine 

Saturday 

0800-1100 Download/upload 
Sunday 

0800-1100 Download/upload 

1900-2100 Go online for office hours 



Like their students 
instructors access SMART 
via the task list; they 
can see grades on lessons 
that are computer scored, 
pick up assignments to 
grade, post grades, 
receive questions & 
comments & post answers 
to them; students who are 
not getting online either 
because they have fallen 
behind or are having 
computer problems must be 
contacted by phone 

In SMART, the instructor 
has the opportunity to 
consult with experts or 
written resources before 
answering a question 

The time needed for 
downloading and uploading 
will vary by volume 



SMART instructors do both 
class admin (grading, 
etc.) and computer admin 
tasks 

Synchronous activities 
must be scheduled around 
students 7 other activities 
& prompt feedback is 
critical in SMART 



To ensure timely turn- 
around instructors must 
be on the computer daily 



Figure 3. A week in the life of a SMART instructor 
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Background 



The literature was reviewed to determine those variables most likely to 
maximize throughput and quality (performance and acceptability) and to 
minimize cost in distributed training. Most of the literature describes 
public civilian education findings in what is commonly called "distance 
education." However, much of this literature is, nevertheless, applicable 
to military, distributed, asynchronous computer-mediated training. The 
following tables summarize the findings of that review. (See Wells, 1990, 
for the full review.) 

In addition, a pilot test was conducted to determine the feasibility 
and logistics of remotely delivering a computer-mediated course for the RC, 
and to begin to identify the important throughput, quality, and cost 
issues. A brief description of this pilot test follows the literature 
summary. 

Literature Review 

Distance education is a worldwide phenomenon. Remote delivery of 
education is popular where students are geographically dispersed or rugged 
climate conditions prevail (for example, in Canada, Norway, and Sweden). 
Third world countries use distance education in an attempt to educate their 
people in the most economical way possible and developed nations, such as 
the United States, provide distance education to adults who cannot easily 
attend traditional universities because of geographic dispersion or heavy 
obligations to career and family. Thus, the principles of distance 
education are derived from a wide variety of applications, both academic and 
industrial . 

While this summary examines variables affecting successful remote 
delivery of education in general, it also considers the effect of these 
variables on a special form of remote delivery, asynchronous computer 
conferencing (ACC). ACC is the computer communications medium on which 
SMART courses are based students in SMART courses communicate with one 
another and with the instructor via a computer conference. Thus, SMART is 
just a specialized form of distance education. Distance education 
principles should apply to SMART in the same fashion as they apply to other 
remotely delivered instruction. 

This literature summary is presented in tabular form and is divided 
into three major sections. Table 1 addresses throughput, which concerns 
variables affecting dropout and rate of progress through a course, quality, 
the variables affecting outcomes such as level of performance and student 
acceptance, is discussed in Table 2. Finally, variables affecting cost are 
shown in Table 3. If a variable affects more than one of these categories, 
it is listed in each category. 
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Table 1 

Variables Affecting Throughout 



Finding 



Imol ications 



References 



Student-to-student 
interaction , decreases 
dropouts, thus enhancing 
completion rates; group 
interaction also facili- 
tates student study 
rates. 

Students with lower 
educational levels are 
more "likely to drop out. 



Group interaction oppor- 
tunities should be incor- 
porated in course design. 



Personal contact with an 
instructor , whether 
student- or instructor- 
initiated, has a positive 
impact on throughput. 

Course completion is 
positively related to 
prompt feedback by the 
instructor. 

The rate of assignment 
completion can help 
predict course comple- 
tion; completion of 
assignments in the first 
one-third to one-half 
of the course is espe- 
cially critical to dropout 
reduction. 

Increasing the variety 
of media lowers the drop- 
out rate. 



Educational level can be 
used to predict not only 
which students are likely 
to succeed, but also which 
students might need addi- 
tional assistance to 
succeed. 

Course design should 
incorporate instructor- 
student interaction. 



Baath 
Pentz 



Use a full range of media 
to enhance throughput. 



(1982); 
and Neil 



(1981) 



Wells (1989); 
Woodley and Parlett 
(1983) 



Course materials should 
be designed with type 
and frequency of feedback 
in mind. 

To facilitate completion, 
increase the frequency 
and decrease the length of 
assignments. 



Coldeway, MacRury, and 
Spencer (1980); 
Woodley and Parlett 
(1983) 



Coldeway (1980); 
Rekkedal (1982) 



Wong and Wong (1979); 
Woodley and Parlett 
(1983) 



Pentz and Neil (1981); 
Woodley and Parlett 
(1983) 
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Table 1 



(continued) 



Finding 

Student dropout is higher 
under conditions of 
individual rather than 
instructor pacing . 

Larger class size can 
lead to less feedback, 
slower response times, 
and higher student 
attrition. 



Imp! ications 

Externally set deadlines 
should be used in dis- 
tance courses to facili- 
tate completion. 



References 



Coldeway (1982); 
Morris, Surber, and 
Bijou (1978) 



Twenty-five students is Hiltz and Turoff 

probably a reasonable (1978); Manock 

class size, but this may (1984); Woodley and 

vary as a function of Parlett (1983) 
media selection and 
course content. 
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Table 2 

Variables Affecting Quality (Per formance and Acceptability) 



Finding 



With respect to student 
motivation : students 
using ACC reported 
trying to do a more 
thorough job on assign- 
ments because other 
students would have 
access to them; motiva- 
tion plays a role in 
both user satisfaction 
and in acquisition and 
retention of information. 

Ready access to a 
computer is almost a 
pre-requisite for suc- 
cessful performance in 
ACC; quality of access 
(quiet location, etc.) 
also impacts performance. 

Learning skills and 
strategies impact per- 
formance; cognitive 
processing skills (i.e., 
logically connecting 
ideas) are more impor- 
tant than mechanical 
skills (i.e., memoriza- 
tion); students do not 
always use the materials 
as intended by the 
designer. 



Implications 



References 



Motivation may be the 
most important variable 
in util ization of a 
computer conferencing 
system—even a well- 
designed system will not 
be used without positive 
motivation. 



Each student should have 
a computer that can be 
set up in a convenient 
location. 



Instructional materials 
should be designed to 
encourage depth of 
processing; course 
designers should use 
methods to ensure that 
students proceed in the 
proper sequence. 



Baath (1982); Hiltz 
(1986); Kerr and 
Hiltz (1982); Mason 
(1988); Vallee, 
Johansen, Lipinski, 
Spangler, Wilson and 
Hardy (1975) 



Harasim (1986); 
Kirkwood (1988); 
Vallee, Johansen, 
Randolph, and Hastings 
(1974) 



Marton and Svensson 
(1982); Rumble and 
Keegan (1982) 
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Instructors must adopt a 
facilitative role in ACC 
and must take care to not 
dominate the interaction; 
specialized instructor 
training which emphasizes 
the differences between 
distance and face-to-face 
training and which 
provides skills needed in 
the distance environment 
is needed. 



Coldeway (1980); 
Holmberg (1981); Kerr 
and Hiltz (1982); 
Rekkedal (1982); Smith 
(1988) 



Incorporate group acti- 
vities to enhance student 
acceptance. 



Harasim (1986) 



i'J Students should be re- 
Wfc .i k quired to spend a maximum 

io of 10 hours per week on a 
course; course duration 
should be set accordingly. 



Kaye (1981) 



Hiltz (1986) 
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Table 3 

Variables Affecting Cost 



Finding 



Imol ications 



References 



Cost per student increases 
if more personalized 
feedback is "provided. 



Development costs to 
produce high Quality 
materials are high, and 
may exceed those for 
resident training; use 
of existing materials 
reduces costs, but 
rarely produces good 
self-instructional 
materials. 

Media selection will 
impact costs, with some 
media being cheaper to 
develop than others 
(i .e. , a one-hour text 
assignment will cost 
less than a one-hour 
interactive video) . 

The most cost-efficient 
distance education system 
is one with few courses 
and large class sizes 
and in which classes are 
not revised frequently. 



Personalized feedback, 
although costly, enhances 
both feedback and perfor- 
mance. 

High qual ity materials 
enhance both student 
motivation and perfor- 
mance; materials should be 
produced specifically for 
the distance environment. 



Rumble (1988) 



Rumble (1988) 



Use of a variety of media 
enhances throughput, so 
media selection should 
not be driven solely by 
cost. 



Some MOSs may involve 
relatively few soldiers, 
making it difficult to 
achieve an economy of 
scale; doctrinal changes 
may require frequent 
updates . 



Rumble (1988); 
Sparkes (1984) 



Rumble (1988) 
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In summary, the research reported suggests that quality instructional 
design and responsive implementation (including instructor performance) may 
often override student constraints (like time pressure) which would 
otherwise lead to dropout (Woodley and Parlett, 1983). Thus, the primary 
responsibility for student retention and performance may rest with the 
quality of an institution's instructors, courses, and support services. 
Hence, a great deal of attention was devoted to these issues in the current 
project. 

Pilot Project on Using SMART for Delivery of RC Training 

Pilot research was conducted to determine whether SMART had enough 
potential for RC training to warrant more detailed investigation. Results 
of this project are reported here, since they served, in part, as the basis 
for the decisions made in the implementation test. (See Richards, Hahn, 
Phelps, Blackman and Schurman, 1989, for a complete report.) 

The pilot project used Asynchronous Computer Conferencing (ACC) to 
augment Phase III of the U. S. Army Reserve Forces School version of the 
Enqineer Officer Advanced Course (EOAC). The phase consisted of seven 
engineer subcourses offered through the Army Correspondence Course Program 
(ACCP). Topics included technical subjects such as soils Engineering and 
Military Bridges. Students in the pilot test used computers to communicate 
with the instructor and each other to discuss the content presented in the 
ACCP materials and additional requirements developed by the instructor. The 
experimental course ran from November 1986 to June 1987. 

The course management function of SMART was limited primarily to 
providing feedback on online activities. This feedback, however, was 
certainly more rapid and frequent than that afforded by ACCP. Students were 
required to interact directly with the computer conference host computer; 
SMART did not automatically prepare data for uploading or sort downloaded 
data for them. 

Research questions . The research questions addressed in the pilot 
project were specific and limited: 

Can the throughput and quality of Army correspondence courses be 
improved by the addition of ACC? 

What are the costs, logistical requirements, instructional and/or 
motivational methods (such as computer hardware/software and 
instruction methods) necessary for implementation of ACC for RC 
training? 

"he Throughput/Quality issue was examined by comparing two versions of 
Phase III, EOAC In the control version, participants took the module via 
correspondence according to ACCP guidelines. In the experimental versTon, 
the ACCP materials were also presented, but they were augmented by ACC. The 
dependent measures for this comparison were the percent of students 
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completing the course, the speed with which they completed (throughput), and 
student performance (quality). 

To examine the Cost/Logi sties/Implementation issues, the experimental 
version of the course was analyzed as a case study. The activities of the 
students and the instructor were documented with special attention to cost 
and logistic factors and to the results of several instructional and 
motivational manipulations used throughout the course. 

lessons learned from the pilot test . While many of the results 
relating to throughput and quality were not statistically significant, they 
did suggest that further investigation of computer delivery of RC training 
was warranted: more students completed the course and in less time and 
there was a promise of higher level of knowledge gained and retained with 
ACC than with ACCP alone. Specific conclusions were: 

(1) The experimental group showed higher rates of completion 
(percentage of subcourses completed) throughout the class than did 
the control group; 

(2) The experimental group also had a higher percentage of completion 
of the course as a whole than did the control group; of the 
original experimental group, 36% completed the course compared to 
27% of the control group; of the starters in each group, 64% of 
the experimental group completed vs 54% for the control group; 

(3) The experimental group students who completed the course did so 
earlier than the control group completers; there was a 
statistically significant difference between the groups on number 
of days required to complete the course; there was also a 
difference in the number who completed the course on time (by the 
planned deadline) for each group, as compared to those who 
finished by the final deadline; and 

(4) The on-time finishers in the experimental group did better than 
the control group finishers on both post- and retention-tests; 
late finishers in the experimental group performed slightly better 
than the control group on the post-test and did better on the 
retention test; however, performance differences were not 
significant since ceiling effects were evident in the scores for 
both groups. 

As shown in Table 4, costs for the computer -augmented course fall 
between those for correspondence and resident training. If this result is 
coupled with the improved throughput and quality (over correspondence) 
discussed previously, it appears that the question of ACC-medi ated, remotely 
delivered training warrants further study. 
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Table 4 



Cost Comparison for Correspondence, Correspondence plus ACC, RD School, and 
Resident School Training for Phase III of EOAC 



Training Method 


Cost per Soldier 


Correspondence 
RF School 
Correspondence 
plus ACC 
Resident School 


$48 
$3335 

$4195 
$11585 



Data for non-ACC training provided by TRADOC Headquarters or the 1155th RF 
School . 



The logistics and implementation issues considered dealt primarily with 
usability and acceptability of the ACC system. Most soldiers, regardless of 
previous experience, were able to set up and operate their computers. In 
general, soldiers in the experimental group liked using the computers. On a 
five-point scale, soldiers rated SMART presentation at 3.8, below the rating 
of 4.0 for resident school, but above the rating of 3.1 for correspondence 
courses alone. ACC was rated almost the same as RF Schools (3.8 vs 3.7, 
respectively) . 

Logistical lessons learned included: 

(1) Keep both hardware and software as simple as possible; soldiers 
are not computer experts and problems in these areas will be extremely 
frustrating and may lead to dropouts; 

(2) Support communications, such as a telephone hotline, to assist 
soldiers who are having difficulties are essential; 

(3) Take into account travel, odd work schedules, and family plans of 
part-time students by allowing some flexibility in scheduling; 

(4) Individually accomplished activities are more accommodating to 
personal schedules than are group activities; 

(5) Time requirements must be reasonable for the part-time student; 

(6) Conferencing must be an integral part of the course requirements, 
not a sideline to a course primarily conducted in some other fashion; and 

(7) Participation should be integral to completion of course 
requirements; soldiers should be judged on the basis of completion of 
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learning activities (such as participating in a group exercise) rather than 
strictly on the number of comments entered. 

Rationale 

Based on the encouraging results of the pilot project a more rigorous 
implementation test of SMART was conducted to study throughput performance, 
acceptability, and cost of a SMART course. The general approach was to 
develop a "best shot" course, that is a course which was designed to 
maximize throughput, performance, and acceptability. The 1 iterature on 
remote training and distance education, general principles of instructional 
Ses?gn and experiences from the pilot project were used to determine the 
specific factors investigated within the course and to guide the course 
design . 

Course design parameters were specified at the outset of the course, 
with no a priori intention to systematically alter them as the course 
progressed^o^ever, since it is recognized that course effectiveness is 
largely governed by good course design, we did allow formative changes in 
these parameters as need dictated. These formative changes were documented 
as part of the "lessons learned" from the project. 

Several independent variables (including group interaction, pacing, 
instructor characteristics, and type of course requirements) were 
y tematUany manipulated during the course Again, formative changes were 
allowed as needed, to maximize success. (A full description of both the 
course design parameters and the independent variables is contained in the 
"Method" section.) 

Because the results of the pilot project were encouraging and because a 
major effort, guided by established literature, went into the course design, 
we felt that SMART was ready for a summative test. However, it was also 
felt that learning to optimize course design was an important outcome of the 
Project Thus, formative analyses were also a valuable source of data to 
add to our knowledge base on course design for the distance learning 
environment. 

Method 
Participants 

Students in the SMART class were recruited from a pool of soldiers who 
(1) needed to take Phase III of EOAC and (2) had already taken Phase I. All 
interested students who were available in the desired time frame were 
permitted to take the course. 

Fourteen students began the course. These students were all 
Lieutenants and Captains and all had at least some college education (12 of 
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the 14 were college graduates), although only about half majored in 
engineering. Their average age was 34, and only one was female. 

The control group consisted of RC officers who took Module 6 of EOAC in 
residence at the Engineer School during fiscal years 1987, 1988, and 1989. 
Data on test (n=370), homework (n=171), and course culminating practical 
exercise performance (n=165) were collected and pre- and post surveys were 
administered to a subset of the RC students (n=49). 

Material s 

Course Description 

The course consisted of Module 6 of the Engineer Officer Advanced 
Course (EOAC) as it was taught by the USAES (U.S. Army Engineer School) in 
1987-88. EOAC is a mid-level course (i.e., between senior service school 
and technical MOS training) for officers in the grades of lieutenant and 
captain. Engineer officers must complete EOAC before they can be promoted 
to major. It was felt that the participants in this group would be mature, 
be committed to the Reserve Components, and be more apt to have either 
personal computer experience ur the ability to learn quickly to use 
computers. Additionally, since 68% of all Army engineers are in the RC, 
there was a clear potential benefit of enhancing RC training within the 
Branch. 

Further, the Program of Instruction (POI) for EOAC includes a mix of 
technical and leadership objectives. There is a stated instructional goal 
for the development of group skills through small group instruction and 
group interaction, an ideal application of ACC. 

When taught in residence, Module 6 of EOAC consists of a mix of 
classroom lectures, lead through practical exercises, computer-assisted 
instruction, small group instruction, and a culminating practical exercise. 
The module is taught over a two week period. Both RC and AC students 
participate in the class. In the time. period during which data were 
collected for this project, 55% of all resident students were RC. However, 
the AC/RC mix in resident programs is highly variable, and depends on 
training budgets and slots available. 

Nine topics were taught in Module 6: 

-- Rear Operations 

~- Airfield Damage Repair 

Military Petroleum Pipelines 

Allied Presentation (Australia) 

Asphalt Production 
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Army Writing Program 
Flexible Pavements 
Roads and Airfields and 
Bridging. 



At the end of the module, a course culminating practical exercise is 
conducted. This exercise, which is referred to as a 'capstone by the 
Engineer School, is a group activity which requires students to integrate 
technical information learned throughout the module and to coordinate 
efforts to share limited resources. Thus, students are presented with a 
series of construction directives which require technical solutions to 
bridging, roads and airfields, and similar problems. Additionally, students 
are qiven a time frame over which their designs must be able to be 
implemented, and a list of personnel and equipment avail able to ""Pigment 
them. Group members must negotiate with one another for these resources so 
that both individual directives and the entire project can be completed 
within the stated schedule. The culmination of the exercise is a briefing 
to a field grade officer, in which design alternatives, scheduling 
constraints, and recommendations are presented. 

Course Design 

The Resident School structure and EOAC contact were adopted, as they 
were iudqed to be the best training the Army had to offer. Thus, materials 
from fheresiSent course were adapted for use in the SIWPJ environment. 
Procedures for accomplishing such an adaptation are fully fussed J" 
"Recommendations for Course Development and Implementation of SMART Courses 
(Hahn, Harbour, Schurman, Daveline, and Wells, 1990.) 

Course desian was guided by the literature and the results of our pilot 
project. Details of the course design parameters are given below. 

Ti me reouirement s and course duration . Students were permitted to 
participate in the course only if they would agree to spend a minimum of 
eight hours per week. Total course duration was set to accommodate an eight 
hour per week schedule. 

Cla ss size . Class size was dictated by student availability However, 
different types of activities were structured to involve different numbers 
of students For example, all students participated in classroom 
discussions! but students were divided into groups of four to seven to work 
on group activities. 

Cou rse structure . The course was designed to minimize student anxiety 
by be ginning with eas y, short lessons. Generally, lesson activities were 
kept as sho rt a s possible. Often, several activities were used to cover the 
material presented in a single lecture at the resident course. 
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Variety of media . Several different media (including paper-bused 
materials, CAIs, storyboards, video-tapes, spreadsheet exercises, and 
synchronous and asynchronous discussions) were used in the course. The main 
constraint imposed in media selection was that all media had to be suUable 
for in-home presentation. 

learning skills and strategies . A listing of the tasks and the 
sequence in which, they should be completed were used to help assure that 
students used materials in their logical order. Further, structured tasks, 
notetaking aids, "mandatory" time on task, and frequent checks on 
understanding were used to increase cognitive processing. 

Feedback . Many of the learning activities were designed to provide 
immediate feedback during or upon completion of the activity. For example, 
performance feedback was provided throughout CAI lessons. Paper-based 
activities and storyboards were usually followed by a computer-scored quiz. 
Some activities required instructor grading; instructors were required to 
provide 24 hour turnaround on grading these activities. Further, 
instructors were expected to address all student questions or comments 
within 24 hours. 

Motivational strategies . A variety of motivational techniques were 
used in an attempt to enhance motivation during the course. These included: 

(1) Providing feedback to students regarding their progress in the 
course. The instructor used a set of decision rules for 
determining how close to the schedule students were keeping. 
Students were given messages ranging in tone from "You're right on 
schedule. Keep up the good work" to "You're more than 25% 
behind. We need to write a 'contract' for you to catch up. If 
you don't catch up, you'll be dropped from the course." 

(2) Offering bonus points for on-time completion of the course. 

(3) Reducing the maximum point value for late assignments to 75 rather 
than 100 points. 

(4) Posting an honor roll of each week's "best achievers." 

(5) Providing feedback on participation in team projects. Students 
were shown how their participation in group exercises compared to 
that of other team members. Further, each team was shown how 
their team's performance compared to that of the other team. 

(6) Implementing a buddy system in which two students were assigned as 
partners to assist one another throughout the course. 
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(7) Promising to send letters of commendation and/or appreciation from 
the Assistant Commandant of the USAES to students after course 
completion. The type of letter sent was determined by the 
student's course participation. 

Instructor characteristics . If the Army were to implement SMART for RC 
training, it is possible that part-time members of the RC would be employed 
as instructors. Our pilot project experience, however, was only with a 
full-time instructor. Thus, we had no information as to whether and how 
well (in terms of the outcome measures) part-time instructors could be used 
in the SMART environment. 

Hence, in the implementation test, both full-time and part-time 
instructors were used. Five Reserve Component officers were assigned as 
instructors. One of these was full-time and served as the primary 
instructor, backing up the other instructors as needed to ensure the Quality 
of the instruction provided to the students. He had successfully completed 
the EOAC course and had prior experience with SMART as the instructor in the 
pilot project. 

The other four instructors were assigned part-time in various 
capacities at different times during the course. The roles assigned 
included instructor, team leader, and course culminating practical exercise 
moderator. All instructors received 40 hours of training in SMART 
instruction prior to the beginning of the course. 

Support communications . A free hotline was set up for the duration of 
the course. Students experiencing difficulties could call this hotline for 
help 24 hours a day. When the hotline was not staffed, messages were 
recorded and help was provided as soon as possible. Further, students had 
free telephone access to the primary instructor for the duration of the 
course. He provided help on hardware and software problems, course content, 
and personal difficulties. 

Procedure 

As is shown in Figure 4, the course was implemented in three phases. 
Several variables, including (1) group interaction opportunities (individual 
vs group instruction), (2) pacing (individual self-set schedule vs 
externally imposed group paced deadlines), and (3) type of course 
requirements (content orientation in both Phases 1 and 2 vs process 
orientation in Phase 3), were altered across these phases. Content oriented 
course materials are those which focus on the acquisition of topical 
material. Process oriented course materials are those which focus on the 
application of content knowledge, as in the course culminating practical 
exercise. 
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Phase 1 



Phase 2 



Phase 3 



Individual 
Self-Paced 
Instruction 



Small Group Pacing 
and Instruction 



Course 
Culminating 
Practical 
Exercise 



Figure 4. Three phases of SMART 



Phase 1 had the following characteristics: 

(1) all activities were designed to be completed individually, with no 
group interaction required; 

(2) students were permitted to work at their own pace as long as they 
conformed to the minimum time requirements of the course; and 

(3) course materials were content-oriented, teaching the following 
topics: Rear Operations, Airfield Damage Repair, Military Petroleum 
Pipelines, the Allied Presentation, and Asphalt Production. 

The characteristics of Phase 2 were as follows: 

(1) both individual and group activities were used; 

(2) students were given a deadline for each topic; further, group 
activities required that students be up to date with their course work to 
participate, thus, serving as a pacing aid; 

(3) course materials were content-oriented, teaching the Army Writing 
Program, Flexible Pavements, Roads and Airfields, and Bridging. 

Phase 3, the course culminating practical exercise described 
previously, consisted of: 

(1) a group activity with 

(2) an externally imposed deadline and 

(3) a process orientation. 

The SMART software was tailored to the functional requirements of each 
phase. For example, in Phase 1, students were allowed to have social 
contact with one another in the break room, but content-oriented 
interactions were limited to those between the student and the instructor. 
In Phase 2, content-oriented interactions were unlimited. A full 
description of the software implementation for each phase can be found in 
Appendix B. 
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As shown in Table 5, summative data were collected during each phase of 
SMART . In addition, summative data were collected on Resident RC students 
who took EOAC during the time period that the SMART course was being 
developed and implemented. 

Analysis and Results 

Summative Findings 

Two types of summative comparisons, external and internal, were 
conducted. % External analyses compared throughput and performance data from 
the SMART test vs data from the Engineer School resident version of 
Module 6. Internal comparisons looked at throughput, performance, and 
acceptability (survey and interview) data collected in the two 
content-oriented phases of the SMART implementation test. Comparisons were 
also performed among the phases to determine the impact of group 
interaction, pacing and deadlines, and type of course requirements. Survey, 
interview, and online data were used in those comparisons. 

External Comparisons 

Throughout . Clearly, resident training is superior to SMART training 
with respect to both the duration of time needed for training and the 
percentage of students who complete the training. Completion of the 
equivalent amount of content material to that taught in our course is 
accomplished in two weeks at resident school. SMART administration took 31 
weeks. Further, the drop-out rate at the resident school is quite low (5% 
or less) . 

Both phases of SMART training had a fairly large percentage of students 
complete training. Of the 14 students who started Phase 1, only one 
dropped, for a drop-out rate of 7%. In Phase 2, two of the starters, or 
15%, dropped. Another two students stayed in the course for the entirety of 
Phase 2, but failed to take all of the exams required for course 
completion. Even counting these two students, though, 64% of those who 
started the course completed it. 

Performance . Four types of performance comparisons were made between 
resident and SMART students. These included test performance, homework 
performance, course culminating practical exercise performance, and 
perceived skill ratings. Test performance was the most difficult of these 
comparisons because it was not possible to equate testing conditions for the 
two groups. 

Students in the SMART course were given test questions pertinent to 
each subcourse immediately following each topic. These questions were the 
same as had been used in the resident end of course test at the time we 
began course conversion. However, by the time data collection began for 
resident school students, the resident course tests had been changed. The 
new test covered the same learning objectives as the previous test, but 
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Table 5 

Summative Dependent Measures 



Measure 



How Collected 



How Used 



Course duration 



% Completion 



End of course test 



Homework scores 



Course culminating 
practical exercise 
scores 



Skill ratings 



Measured length of time 
needed to complete course 
for Resident vs SMART and 
length of time to complete 
oach phase in SMART 

Measured number of 
students completing vs 
number of students 
starting 

SMART students completed 
a subcourse test at the 
end of each topic; 
Resident students were 
tested on all subcourses 
at the end of the module 

Both SMART and Resident 
students completed 
homework assignments 
throughout the course 

Both SMART and Resident 
students completed a 
graded culminating 
practical exercise at 
the end of the module 

Both SMART and Resident 
students completed a 
pencil and paper survey 
of their perceived skill 
on each Module 6 topic 
before and after taking 
the course; pre-post 
comparisons indicated 
perceived learning 



Resident vs SMART 
comparison and 
Phase 1 vs Phase 2 
comparison within 
SMART 

Resident vs SMART 
comparison 



Resident vs SMART 
comparison 



Resident vs SMART 
compari son 



Resident vs SMART 
compari son 



Resident vs SMART 
comparison and 
comparison of 
Phases 1 and 2 
within SMART 
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Table 5 (continued) 



Measure 

Scores on each activity 



Percei ved acceptabi 1 i ty 



Reasons for dropping 



Perception of group 
interaction 



Type and content of 
online interactions 



Perception of pacing 
and dead! ines 



How Collected 

For SMART students, each 
learning activity was 
graded 

SMART student opinions 
on course acceptability 
were polled using a 
post-course survey, 
interviews, and 
comments placed online 
by students 

Survey administered to 
SMART students who 
dropped the course 

SMART student opinions 
on group interaction 
were polled using a 
post-course survey, 
interviews, and 
comments placed online 
by students 

Onl ine dialog was 
analyzed to determine 
the amount of student- 
student, student- 
instructor, and 
instructor-student 
dialog, and the content 
of those interactions 



How Used 

Comparison of 
performance in 
Phase 1 vs Phase 2 

Comparison of 
acceptabil ity of 
Phase 1 vs Phase 2 



As an indicator 
of acceptabil ity 

Comparison of 
group vs individual 
learning 



Comparison of 
interaction 
patterns given 
group vs 
individual 
learning modes in 
Phase 1 vs Phase 2 
and content vs 
process orientation 
of Phase 2 vs Phase 3 



SMART student opinions Comparison of 

on pacing and deadlines individual self- 
were polled using a pacing vs group 
post-course survey, external pacing 
interviews, and 
comments placed online 
by students 
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Table 5 (continued) 



Measure 



How Collected 



How Used 



Actual vs expected 
completion time 



Perception of content 
and process oriented 
activities 



For each topic, a record 
was kept of when each 
student completed vs 
when completion was 
expected given an 8 hr 
work week or deadlines, 
as appropriate 

SMART student opinions 
on content and process 
oriented activities were 
polled using a post- 
course survey, interviews, 
and comments placed 
onl ine by students 



Comparison of 
individual self- 
pacing vs group 
external pacing 



Comparison of 
content vs process 
learning activities 
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different questions were used. Hence, SMART subcourse scores and resident 
school end of course scores are similar with respect to the learning to 
testing interval (i.e., both are post-tests administered immediately 
following content presentation) but are dissimilar with respect to the 
specific questions asked. Because the questions on both exams do test the 
same learning objectives, however, this is probably the best comparison 
available. 

Based on this comparison, there is no significant difference in the 
test performance of SMART and resident students (F (1, 377) = 1.80, 
d < 0 18) * In fact, mean scores of the SMART students are higher (although 
not significantly so) than those of the resident students (91.96 vs 86.44). 

Further, 100% of the SMART completers achieved passing scores on their 
criterion test on the first attempt; only 93% of the resident school 
students in our sample achieved a passing grade (70 or better) on their 
first try at the exam. 

A multivariate analysis of variance was performed on the homework 
scores of the two groups. This MANOVA showed a marginally significant 
difference in performance of the two groups on their overall homework scores 
(Wilk's Lambda = 0.95, F = (4,175) 2.302, p < 0.06). However univariate 
analyses of variance on the individual homework assignments showed no 
significant differences between the two groups. Table 6 shows mean homework 
scores for the two groups. 

Course culminating practical exercise performance was also compared for 
the two groups. An analysis of variance was conducted, using the practical 
exercise scores as the measure of interest. No significant differences were 
found on the grades of the two groups, which averaged 90.44 and 89.99 for 
SMART and Resident School, respectively. 

Both SMART and resident students also provided ratings of their skill 
level in the content areas both before and after taking the course. 
Pre-course skill ratings did not differ between the two groups. 

Pre-post differences were analyzed to determine the perceived 
performance benefits of the two groups. A MANOVA performed on these 

Table 6 

Homework Scores 



Topic 



Flexible Pavements 
Bridging 

Roads and Airfields 
Pipe! ines 



Resident 
Mean 

91.73 
91.31 
95.78 
95.39 



SMART 
Mean 

83.33 
82.56 
89.70 
88.33 
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lally significant (Wilks Lambda » 0.49, F 
Univariate analyses of variance showed 
,r fferences between the two groups with respect to 
• * on rear operations (F ( 1 > 42 ) = 5.73, £ < 

(F (1,42) - 13.40, fi < 0.0007), pipelines (F 
uHble pavements (F (1,42) = 8.67, £ < 0.0052), 
.2) = 15.42, £ < 0.0003). In all cases, SMART 
^earning benefit than resident school 
r. • ^ and standard deviations of the pre-post 



R atings 

Resident Mean (SD) SMART Mean (SD) 



rained by subtracting pre-course ratings from 
- . Ratings were based on a 1-5 scale. 



• rocedure" section, the course was divided into 3 
acing, group interaction, and content, the 
oMt differences found among those phases. 

' to the time required for completion, as 
n time, Phase 2 was clearly superior to 
nes longer than expected to complete Phase 2 
, cted to complete Phase 1. As will be 
he use of deadlines is primarily responsible for 
n Phase 2. 

*ws the average scores on each graded activity 
, ugh it would appear that students performed 
• ' onted in Phase 1 than those presented in 
- a function of lesson content than any 
selves. Bridging, on which scores were 
red to be the most difficult topic addressed in 



0.136 (1.13) 

0.409 (1.13) 

0.452 (1.13) 

1.250 (1.08) 

0.932 (1.07) 

0.500 (1.08) 

0.809 (0.99) 



1.125 (0.64) 
1.875 (0.83) 
1.500 (1.60) 
1.875 (0.99) 
2.125 (0.99) 
0.625 (0.92) 
2.250 (1.17) 
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Table 8 

Scores on Graded Activities 



Activity Mean Score Stand_ard 

ft CT -' vl - 1 Devi ation 

Phase 1 c 00 

Rear Operations Exam 96.67 -J- 

Airfield Damage Repair Exam 97.78 

Pipelines Homework 88-33 • 

Pipelines Exam 94 78 8 69 

Asphalt Production Exam 93.33 

Phase 2 07 co 

Bridging Homework 82.56 «• 

Bridging Exam J3. 56 • 
Roads and Airfields Homework 95.78 

Roads and Airfields Exam 98.56 



2.24 



KUdUi dliu nil 1 iciuj n or 

Flexible Pavements Homework 83.33 *-^» 
Flexible Pavements Exam 88.33 . 

Students provided self-ratings of their perceived learning Prior to 
beginning the course, students rated their level of skill in each content 
aria After the course was completed, these ratings were repeated. The 
difference between the ratings was taken as an indicator of Perceived 
learning benefit. Difference scores on Phase 1 topics were compared to 
hose on Phase 2 topics. No significant difference was found in student 
perception of learning benefits between the phases (t (54) - -0.2354, p. 
0.82). 

Acceptability . Based on post-course survey jesponses, interviews, and 
online comments, SMART students seemed to prefer Phase 2 over Phase 1 
Secause of the group interaction opportunities it afforded Students also 
pxnressed a preference for SMART training over that Provided in 
correspondence or RF Schools. Some students even felt that SMART competed 
favorably with resident school. 

Students who left the course were questioned about their reasons for 
dropping. Sixty seven percent cited personal reasons, such as family 
commitments or the need to complete the course in a certain time frame, as 
the reason that they dropped the course. Students who were 
dissat fied with the course itself were generally concerned over the time 
commitment required, rather than technical aspects of the instruction. 

Group, interaction . A great deal of data was c jll e ^^ v ^ d ;" g ti ^ e tv 
number and types of interactions which occurred in the '"dividual activity 
mnHP (Phase 11 vs the group interaction mode (Phase 2). In general, 
interactive atterns changed between phases, reflecting the group emphasis 
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in Phase 2. Further, students reported positive effects of group 
interaction. Detailed results follow. 

Interaction patterns definitely differed between the phases. In 
Phase 1, communication between students was only allowed in the break room. 
As Figure 5 shows, however, even in this forum only 25% of the communication 
was between students. Most communication in the Phase occurred between a 
student and the instructor. 

In Phase 2, student to student communication in the break room 
accounted for about 22% of the total traffic, similar to the proportion seen 
in Phase 1. However, this figure does not truly represent the amount of 
student to student communication that took place. As can be seen from 
Figure 6, in the team rooms nearly 60% of the interaction that took place 
occurred between students. The interaction patterns observed during Phase 2 
make sense in light of the functions of the various rooms. The team rooms 
were set up as a place for groups of students to accomplish given tasks; 
conversely, the break room was an open forum for all speakers and the class 
room was oriented toward student/instructor interaction, since it was the 
place where students were to go to ask questions. 

100 



80 




Instructor/Student 



Student/Student 



Student/Instructor 
Interaction Type 

| Percentage 
Note An entry is any comment placed online. 

Figure 5. Types of interaction in break room communication during Phase 1 
(n=132) 
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B Team Percentage £v] Other Percentage 

Note. An entry is any comment placed online. 
Team percentage = team room entries; Other 
percentage = entries in all other rooms. 

Figure 6. Types of interaction during Phase 2 (n=924 and 454 for team and 
other, respectively) 



The content of the classroom interactions differed between Phases 1 and 
2 Fiqure 7 shows the content of these interactions for the two phases. In 
Phase 1 the majority of the discussion centered around solving 
content-related technical problems, motivation, and computer questions and 
problems. In Phase 2, the emphasis shifted to techmcal discussion 
requests for help on technical subjects, and responses to those requests. 

These changes partially reflect the shift from individual to group 
emphasis between the phases. In Phase 1, students solved technical 
exercises on their own and discussed them with the instructor in the 
classroom In Phase 2, technical exercises were usually conducted as a team 
effort; thus, the discussion surrounding the solutions to technical problems 
occurred in the team rooms. 

Further, in Phase 1 the instructor worked with individual students to 
foster motivation. In Phase 2, emphasis was placed on team spirit as a 
motivator and motivational messages were nore likely to be delivered in the 
team rooms than in the more individual -oriented classroom. 

When students were asked about their preferences for group or 
individual work 43% of the respondents said that they preferred team work, 
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Type of Content 

Phase 2 Percentage 



| Phase 1 Percentage 
Note: An entry is any comment placed online. 

Figure 7. Content of classroom interactions for Phases 1 (n=206) and 2 
(n=251) 



while 29% said that they preferred individual work. Clearly, group 
activities are accepted favorably. 

Further, all respondents either agreed or strongly agreed with the 
statement "Working in teams motivated me to participate in the class" 
attesting to the positive impact of group interaction on throughput. One 
student commented "Team work positively impacted motivation as compared to 
the individual section." Another said: "I'm enjoying the team exercises, 
and I'm learning faster and more thoroughly by being able to share everyone 
else's knowledge and expertise." 

Pacing Two different pacing issues were addressed simultaneously: 
the effects of using individual vs group activities as pacing aids and the 
effects of deadlines. General results were: (1) group activities require 
more calendar time for completion, thus, slowing the course; however, they 
also serve as a pacing aid, because students must be caught up with previous 
course work in order to participate; and (2) deadlines are effective in 
pacing students through the course. A detailed explanation of these results 
follows. 



Individual activities can 
asynchronous group activities. 



be completed in less calendar time than 
The longest individual activity planned had 
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an estimated duration of three hours. Based on the eight hour per week 
participation requirement, it was necessary to allot three days tor 
completion of this assignment. Most individual activities were designed to 
be completed in one hour. These were scheduled to be completed in one 
calendar day. 

Time estimates for asynchronous team assignments ranged from 0.5 to 1.5 
hours For asynchronous activities, however, teams were prompted to begin 
work at least one week before the deadline. One average, asynchronous 
activities took six days to complete. 

Two synchronous activities, in which students were required to all work 
at the same time, were each scheduled to be completed in two hours (each 
activity was completed in one day). Actual completion time averaged 3.5 
hours While the synchronous activities were all completed in one calendar 
day they did require substantial lead time so that students could set up a 
meeting time. On average, students began organizing synchronous activities 
nine days prior to their occurrence. 

In spite of the fact that group activities slow the pace of the course 
in calendar time, they may also serve as a paring Students reported 

that they tried to be caught up on their individual activities so that they 
had the knowledge needed to contribute during group exercises: Saturday 
morning I discovered our group project is due Sunday evening and Ken and I 
are responsible for Part 2. My conscience says I am not holding up my end 
of the bargain, so I have worked the entire exercise. Part 1 and 2 to 
follow. But first I had to work the first several parts of segment seven so 
that I may know what in the world is going on." 

Very few other comments were made by students concerning pacing effects 
as a result of using individual vs group activities. A few comments were 
made, however, regarding pacing effects of asynchronous vs synchronous 
activities. Online comments showed a preference for synchronous activities 
since their use allowed students to gain quick closure on an activity then 
move on to something else. However, use of too many synchronous activities 
would result in too rigid a schedule. 

The imposition of deadlines proved to be critical in keeping students 
working close to schedule. Figure 8 shows student progress through the 
course during Phase 1 when the course schedule was self : imposed. The 
"expected completion" line reflects the number of activities that should 
have been completed had students been spending eight hours per week on the 
course. 

Proqress was so poor, that a formative change was made part way through 
Phase 1 and deadlines were imposed. Figure 8 also shows that progress 
through Phase 1 improved once deadlines were set. 

Figure 9 compares progress in Phases 1 and 2. The Phase 1 data reflect 
both the difference between the expected number of days for topic completion 
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Figure 8. Course progress during Phase 1 
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Figure 9. Course progress during Phase 1 vs Phase 2 
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if students were spending the required eight hours per week and the actual 

number of completion days. Based on the positive impact of imposing 

deadlines in Phase 1 demonstrated above, these data must be considered "best 
case" in comparing progress in Phases 1 and 2. 

The Phase 2 data reflects the difference between the scheduled 
completion time given the deadline dates vs the actual completion time. 
Note how closely the schedule was adhered to under the deadline condition in 
Phase 2. In the worst case, Topic 7, students only missed the deadline by 
an average of two days. 

Students felt that deadlines helped them keep pace with the course and 
their acceptance of the deadlines was quite positive. One student commented 
"The best thing you did was to establish deadlines. Once you did that, I 
would log on night after night until I got there or got ahead, and then I 
would take time off, instead of spacing the thing out. My goal was to meet 
the requirement, and then take time to do personal stuff." 

Type of course requirements . Based on a review of the online dialog, 
it is clear that the quality of performance was better on the 
content-oriented activities than on the course culminating practical 
exercise. Both teams got off to a very slow start on the practical exercise 

they spent a large amount of time talking about what they should do, 
rather than actually doing it. Further, both teams had periods of several 
days in which no one did anything! Although students could have been 
working offline, team members awaiting input from them were unaware of their 
progress. 

Figure 10 shows the distribution of online entries dealing with 
technical content (things such as solutions to technical problems, technical 
discussions, and questions) vs those dealing with organization during the 
course culminating practical exercise and during other team activities (both 
synchronous and asynchronous). Data from the two teams were combined, since 
their interaction patterns were quite similar. Note that during the content 
activities, students spent about equal amounts of effort planning and 
performing activities. During the practical exercise, more effort was 
devoted to organization and less to actually performing the task. 

Teams approached both the course culminating practical exercise and the 
content-oriented projects similarly, by having several (usually two) people 
working on the same parts of the problem, verifying each others' work. This 
method did not work well for the students in the practical exercise, 
however, because the teams were small in relation to the amount of work 
required. (See the discussion on class size for more information on this 
issue. ) 

Post-course surveys showed that 57% of the students did not feel that 
their content-oriented group activities prepared them to know how to 
interact during the practical exercise. Post-course interviews revealed 
that 83% of those surveyed did not understand the goals of the exercise. 
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Organization Technical Activity 

Type of Content 



■ Practical Exercise Content Activity 

Percentage iSi-d Percentage 

Note' An entry is any comment placed online. 

Figure 10. Content of interactions during team activities (n=290) vs course 
culminating practical exercise (n=173) 

They expected it to be focused on technical information, as the previous 
group activities had been, and did not expect the process orientation. 

There is no clear cut information available for judging whether there 
is a difference in throughput, in terms of speed of completion, between 
content-oriented and process-oriented activities. Since the course 
culminating practical exercise was a much larger exercise than any of the 
content-oriented activities, it took quite a bit longer to complete. We 
have no quantitative basis for comparing content- and process-oriented 
activities of similar size. However, our feeling based on our observations 
is that process-oriented activities would take longer to complete because 
they require much more coordination among students, as evidenced by the 
increased organizational effort in the practical exercise. This 
coordination is very time consuming in a setting where the coordinator must 
often wait 24 hours or more before receiving an answer regarding the 
suitability of the plan being made. 

In interviews, studenc reactions regarding throughput issues indicated 
that content-oriented activities had a positive impact on completion rate. 
Students were motivated to stay active in the course as a result of their 
participation in team activities. 
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The course culminating practical exercise, coming at the end of the 
module, had no effect on completion rate for the module Students were of 
the opinion that they would complete this one last hurdle no matter how 
difficult. However, judging from the degree of burnout reported, it is 
unlikely that students would have wanted to begin another SMART phase 
immediately following this one. In this respect, the inclusion of 
culminating practical exercises at the end of some or all modules or phases 
may hurt student retention at the course level. 

Based on online comments and interview data, students clearly found 
group content-oriented activities to be acceptable. Representative comments 
include: "You asked us to give our reaction on how we liked it [synchronous 
discussion]. If you don't mind a delayed reaction, let me just say that it 
was great" and "The most enjoyable part^of the whole thing [the course] has 
been the five other folks on this team." 

Comments on the course culminating practical exercise were not so 
itive Many students complained that the workload during the practical 
rcise'was too great and that the capstone briefing was a demoralizing 
experience due at least in part to their lack of knowledge of what was 
expected of them. Other students did feel that the practical exercise was 
o? would Save been a valuable learning experience if they had had some prior 
background in performing capstones and/or giving briefings. 

Summary 

SMART classes compare favorably with resident programs with respect to 
quality and acceptability, and are moderately successful with respect to 
throughput. Proper implementation of group interaction, pacing, and course 
requirements is essential to course success. 

Formative Findings 

Through observation of the course, we were able to learn a great deal 
about the impact of our choices regarding course implementation parameters. 
In most cases, the data gathered are purely informational and do not relate 
directly to the outcome measures. Where data are available regarding 
throughput, performance, or acceptability, however, they are noted as such. 

Student Characteristics 

No significant differences were found between the group of students 
which finished the course and the group which started but did not finish. 
Comparisons were based on demographic data, skills self-ratings, and 
opinions of training methods gathered on the pre-course surveys. 

Drop-out surveys confirm that it was not because of lack of background 
or ability that students elected to leave the course. Rather , students 
mostly reported dropping out because the course did not meet their schedule 
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needs or for personal reasons, such as changing jobs or having other 
commitments on their time. 

Although the differences are non-significant, the survey data show a 
trend indicating that non-engineers may have had more difficulties with the 
course than did engineers. Among the finishers, all of the students had 
engineering-related Army jobs; only 40% of the non-finishers had Army jobs 
which were engineering-related. Similarly, the finishers had a larger 
percentage of engineering civilian jobs, engineering degrees, and computer 
expertise than the non-finishers. Again, however, the differences are not 
significants 

Comparisons were also made between the group of students which started 
the course and the group of students which received preliminary instruction 
about the course and completed the pre-course survey, but did not actually 
start the content-related learning activities. One interesting significant 
difference was found between the starters and the non-starters. The 
non-starters had a significantly more positive opinion of RF Schools than 
did the starters (Satterthwaite t (11.9) = 4.01, p_ < 0.0017). Further, 
although their overall opinions of correspondence and resident courses did 
not differ significantly, the non-starters viewed some aspects of these 
courses more favorably than did the starters. Non-starters had a more 
positive opinion of correspondence with respect to increasing confidence as 
an officer and getting timely questions to answers than did starters. With 
respect to resident school, non-starters had a more favorable impression of 
building professional connections than did starters. This suggests that the 
non-starters may have perceived the availability of other attractive options 
for course completion. 

Further, starters were more likely than non-starters (t (19) = -2.14, p_ 
< 0.05) to report that they understood the learning contract given at the 
beginning of the course. This contract specified what was expected of 
course participants in terms of time commitment, number of online signons, 
and the 1 ike. 

No other comparison between starters and non-starters yielded 
•-ignificant differences between the groups; however a marginally significant 
(p < 0 06) result indicates that non-starters worked more hours of overtime 
per week than did starters. Hence, the non-starters may not have had time 
to participate in the course. 

fime Requirements and Course Duration 

As the course progressed, it became clear that students were not 
ompleting activities as quickly as we had anticipated. Initially, we 
thought that students were not keeping up with their eight hour per week 
time commitment. Online comments and discussions with soldiers, however, 
contradicted this belief. Rather, students reported spending, on average, 
16 hours per week on the course. 
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When queried a, to how their ti« was 255, t M l JS!2 - .i5Sr" B ' 
about eight hours per week ™ ^vHasks " as upload1ng and 
^offiASS^'S-S^^'Th^SSi.l.. had not counted 
. ». i on*- time p<;ti mates . 



this overhead time in our time estimates. 

Students found sixteen hours per week to be too high a workload^ 
are going at a fast pace. It is too fast for me ana i Jhey 

everything else." 

In addition to the eight hour per ^SfaJTast 2l« 
students that they must sign °n to the compu ter a t 1 east t wice P^.^ 

The eight hours, then, had to be ^ s p tn a ^ c °^ended more signons of 
Students found this to be inadequate, and ^coomenaea y . $ 

shorter duration: "You must log on each day^ Logging on P ing> 

not enough. It seems too much water has flown past couldn't 
I would do it [download and upload] once or twice a^week. « 
keep up unless I did it three or four times a week^ Qf 
however will add to course duration, as there is a 
SSsirSive time associated with each log on. 

In analyzing data concerned more" 5 
types of learning activities, we ^""^^^^ • there seems to be 
it^Sl^Xl^^ If- t W es of 
activities) . 

As shown by the standard deviation data given in Figure 11 . storjjto.rds 
and videos, with their inherent pac.n qua| fes are m ,„ 
completed in the tine es t mated t^n a re Wis ana r a ^ ^ 

: h a [fr i ti: fe Thus: t r t typrs\nra%inraarUes P se,ected may have an 
impact on course duration. 



One capstone team had only <™^£^$'3i that there' were 
that, in some cases, °f h f r J° n " ^^'The work of other team members. Even 

Ke'lKS^ - inly a coordinating role ' 
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STORYBOARD PAPER EXERCISE 

ACTIVITY TYPE 

M denotes mean difference between estimated and actual completion 
The distance from x to M is one standard deviation 



PAPER READING 



Figure 11. Means and standard deviations for the difference between 
estimated and actual completion time for different types of activities 

got involved in technical work. At a minimum, it would seem that team 
composition for course culminating practical exercises should be: one 
student for each technical problem, one technical coordinator, and one 
student to check all work. In our case, then, at least six active members 
were needed. 

One student's comment reflects this problem: "There was like four of 
us and we couldn't get enough people getting back on timely things so you 
just couldn't tell what was going on. Everyone said 'Yeah, I'm doing my 
job' and you didn't know if they were or not. You had too many things to 
put time into. Whenever we get into a group activity like that, if one 
person let's you down it really hurts." The difficulty in determining class 
or team size, then, is not one of determining sheer numbers, but, rather, of 
ensuring a minimum number of active participants. In our case, between 
one-quarter and one-third of team members did not participate fully in the 
course culminating practical exercise. 

Variety of Media 

Student comments revealed that there were almost as many opinions of 
the various media as there were soldiers. Two of the media selections, 
however, were universally disliked: reading assignments from TMs and FMs 
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were perceived as boring and difficult; the combination of videotape and 
notetaking guide was disliked because of the need to constantly stop and 
start the film. 

One other point about media arose from student comments: CAI packages 
must be selected so as to accommodate a wide range of answers, particularly 
if students are expected to read from nomographs and/or interpolate. Ihe 
Dackaqe we used allowed only four options for a correct answer. 
Consequently, students were being told that their answer was incorrect if it 
was not one of the four programmed possibilities. Sometimes answers which 
were off by only .001 were marked wrong. Needless to say this was very 
frustrating for the students: "If you didn't get exactly the number in the 
program the answer is wrong. For mathematical problems that's great, but my 
eyeballs are not calibrated exactly like someone else's and therefore m 
wrong When you're talking about interpolating ten numbers within a 1/8 
incS space I feel there should be a little bit of room on either side of 
your answers. 

Finally, students appreciated the use of some paper and pencil 
exercises: "For some reason, I feel more comfortable with paper and pencil 
than putting my answers directly on the screen. I guess that the computer 
has something of a finality about it. But you can always erase the 
Tele H " Student comments'al so stressed the need to ease into computer 
skil s slowly, since the fear of the computer must be overcome: I was very 
intimidated by the computer at first. Not being computer literate, I had to 
counter my fear of doing the wrong thing." 

Following each individual learning activity students filled out a 
short questionnaire on which they were asked about the contribution of the 
media selection to the ease or difficulty of learning. The majority of 
students felt that most media, with the exception of P a P er - bas * d J^ c 
made the technical content easy to understand. For paper-based exercises 
41% of respondents said that they made the content easy to understand 26/, 
said that they made the technical content difficult to understand, and 33/, 
had no opinio^. For all other media, at least 55% of the respondents agreed 
that the media made the technical content easy to understand. 

Iparnina Skills and Strategies 

Comments online and in interviews indicate that students do not always 
read materials as thoroughly as was intended: would skim read a lot 
because of the time crunch so I know what manual to use. I know vaguely 
where things are but I didn't really read it in as technical detail as 1 
could have or maybe should have." 

Further, students were not always sure what points should receive the 
most emphasis: "It would be extremely helpful if the activity description 
included some guidance about what to look for and concentrate on. 
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Finally, students relied heavily on the online listing of SMART tasks 
and became very confused during the course culminating practical exercise 
when it was not present. Although students were mailed a written listing of 
tasks which detailed specific practical exercise activities for them to 
complete, they still reported that they did not know how to proceed. 
Students repeatedly had to be referred to the written instructions by the 
instructor. Thus, changing the format of the cognitive organizers proved to 
be problematic. 

Feedback 

We had set a requirement that instructors must provide feedback and 
answer all questions and comments within 24 hours. In Phase 1, it was only 
possible to track turnaround in days; 55% of student comments and questions 
were answered in one day. This figure is biased on the low side for two 
reasons: the instructor often had telephone contact with the students, and 
answered their questions on the phone prior to putting a response online; 
and the primary instructor waited for the part-time instructors to assume 
their responsibilities -- when they did not, and more than one day elapsed, 
he responded to the students. 

In Phase 2, turnaround occurred within 24 hours in nearly 80% of the 
cases. As shown in Figure 12, most instructor responses occurred either 
within four hours of when the student posted the comment, reflecting mainly 
synchronous discussion during team activities and office hours (periods 
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during which the instructor could be expected to be online), or in the eight 
to 16 hours post-comment time frame, reflecting that the primary instructor 
worked during the day while the students worked at night. 

Student comments indicate that 24 hour turnaround is often too slow, 
especially when the student is working under time pressure and needs to wait 
for feedback to go on. This was a source of frustration for students, as 
indicated by the following comments: "When I would hit a snag, type up a 
message to an instructor or try to call, not being able to get an ^mediate 
response would shut me down for the night" and "I want to get done with this 
tonight and if I try to contact the instructor I will be waiting until 
tomorrow to get an answer. This is the delay in response that constantly 
occurs. I can't expect an instructor to be available all the time, but I 
can't wait to hear from him." 

Obviously, delays in feedback hinder throughput in terms of rate of 
course completion. We had set up our course such that the receipt of 
feedback on a particular activity was a prerequisite for beginning the next 
activity. This caused delays for the students, who wanted to work for 
several hours in a sitting, but were unable to because they could not get 
feedback from the instructor until the next day students suggested that 
the specification of prerequisites be very limited in future SMART courses. 

The quality of the feedback provided, however, was quite good, 
resembling the types of interaction one would expect to see in a 
face-to-face setting. The following is an example of part of a 
student-instructor dialog which took place during the course: 

Student- "I have been working on this PLATO for going on six hours. I 
do not understand what is being explained to me. Rather than just getting 
further behind I'm going on. The calculations on V and M etc. is what I am 
talking about. Do you have any ideas on how I can better grasp this 
theory?" 

Instructor: "If you can be more precise about what you don't 
understand it would be easier to help clarify the problem. The concepts of 
moment and shear involve how the forces act on the bridge foment always 
applies to a lever. It is defined as the force times the length of the 
lever arm. In order for a bridge to support a load it has to be strong 
enough to resist the bending moment. The 'section modulus is a measure of 
a beam's ability to resist bending. A moment has to be calculated for each 
type of load, for the entire bridge, and on a per stringer basis. These 
values plus the allowable bending stress for the type of stringer allow you 
to calculate the required section modulus for the stringer You then^select 
a stringer with a section modulus larger than the calculated value... 

Student comments also revealed that media that offered immediate 
feedback were preferred: "I like the CAIs. The immediate feedback is 
great." 
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Quizzes, too, should provide detailed feedback. Our computer 
administered quizzes were designed to display the student's overall score, 
but did not indicate which answers were incorrect. (This was done so that 
quizzes could be administered more than once; after students completed 
contingency activities they were often referred back to the quiz.) Students 
who received poor grades on quizzes were uncertain as to whether they 
misunderstood the content or had data entry errors: "I am not sure 1 can 
believe a 65% on the quiz and after finishing the CAI I am even more certain 
that I was right. Unless I hit the wrong button. Could you please let me 
know which -questions I missed and what the recorded answers were?" 

Motivational Strategies 

Review of the course dialog, surveys, interviews, online comments, and 
the opinions of the instructor served as the basis for assessing the 
motivational strategies. 

Several of the motivational strategies proved to be quite effective. 
These included: 

(1) Feedback on course progress: Providing feedback on course 
progress seemed to have a positive impact on students' motivation, 
especially in cases where the student had fallen behind. In 
several instances, students entered into a contract with the 
instructor to catch up with the course on a specified schedule. 
Students generally did make an effort to comply with these 
contracts. 

(2) Maximum of 75 points for late papers: This incentive worked well 
in motivating students to get assignments done on time. Only one 
team assignment (out of 12) was late, and this was more a result 
of communication difficulties than lack of effort. (The team had 
the assignment finished, but failed to transmit it to the 
instructor as their "final" answer.) 

Post-survey responses showed overwhelming support for this method: 
85.7% of respondents said that penalizing late assignments was an 
effective motivator. 

Thus, it would seem that penalizing late papers had a positive 
impact on throughput, keeping completion rate on target. 

(3) Feedback on participation in team projects: The instructor felt 
that this technique was effective and caused students to 
participate more than they would have otherwise. Online comments 
support this idea. Students found team participation to be highly 
motivating and also used participation feedback statistics to 
guide their own inputs: "I felt that I had to be right at the top 
in participation in group projects or I was letting down the 
group." 



ERIC 



5. 



44 



On cost-surveys, 100% of respondents said that team membership 
Motivated Them to participate. Further, 57.1% said that knowing 
their participation rate vs that of other team members was 
motivating. 

In light of these findings, it is likely that team feedback 
enhanced completion rates. Teams with members who were not highly 
motivated to participate would probably not have been able to 
rnmolete activities in a timely fashion. Performance may also 
have been impacted! since students knew that other team members 
would be reviewing their work. 

Instructor Characteristics 

The analysis of the appropriateness of using part-time vs full-time 
instructors for SMART courses must be based on a case study approach to our 
exp r e ces during the course. There are so few Quantitative da a on he 
part-time instructors that numerically based comparison of the part- vs 
full-time instructors is generally unwarranted. 

During Phase 1, we had assigned a part-time instructor to serve as the 
"team le der." This role involved counseling students w owe having 
difficulties either with course content or personal P^l^. Only three 
online comments were recorded from this person during th Pjase. Student s 
comments to him went unanswered. Repeated phone calls and v "Js were made 
to the team leader to encourage him to accept his responsibil ities When 
these attempts failed, the formative decision was made j to protect the 
quality of the students' experience with the team leader (a pos tion we telt 
to be crucial) and the full-time instructor assumed the team leader s 
responsibilities for the remainder of Phase 1 and all of Phase 2. 

Two other part-time people were assigned as instructors during Phases 1 
and 2 Of these one was unable to participate due to computer problems 
resulting from an irregular power supply. The full-time instructor assumed 
his duties. 

The other part-time instructor worked out fairly well. He did 
comoetentlv respond to students' comments and questions and did assume the 
admin s?rative duties of the instructor (i.e., grading) during the topics 
for wh ch he was responsible. Unfortunately, students had become very 
reliant on the primary instructor, since he had been the person whom 
thpv had had all their interact ons up to that point. Hence, ne continueu 
to receive questions and phone calls on topics for which the part-timer was 
assigned as the instructor. 

As a result of their experiences with the unreliability of the team 
leader and the part-time instructor who had power problems, student 
accepted of the part-time instructors was quite poor Students had the 

refption ?Lt only the full-time instructor could be depended upon: I 
don't think the instructors participated enough. The only one I really nave 
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a feel for was Keith [the full time instructor]. The other ones [part time] 
didn't seem to get into it too much. A lot of times it was 'Here s your 
instructor' and that's it, so Keith had to pick up." 

The use of a part-time instructor as the course culminating practical 
exercise moderator worked out reasonably well. The nature of the practical 
exercise briefing almost requires that the moderator not be someone with 
whom the students have developed comfortable rapport. Thus, the primary 
instructor would not have been suited for this position. The main 
disadvantage of using this particular part-timer in the capacity of 
practical exercise moderator was that he had not kept up very well with the 
content of the course or the use of the computer conferencing software 
Hence, a good deal of coaching on our part on both technical content and 
computer use was needed as he participated in the briefing This introduced 
many delays into the briefing and made the experience quite hectic for the 
researchers and the primary instructor. 

A small amount of data was collected regarding the time of day during 
which the part- and full-time instructors worked on the course. These data 
are shown in Figure 13. Note that the part-time instructors worked on a 
schedule similar to that of the students -- most work was done at night. 
The full-time instructor worked mainly during the day. This pattern may 
have implications for throughput, in terms of the rate of course 
completion. The schedule used by the part-timers works quite wel 1 if the 
instructor and the student happen to be online at the same time, the student 
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can get a question answered or obtain feedback and move on that same night 
If the instructor and student do not "meet" online, however, turnaround time 
may be increased over that provided by the full-timer who can be online in 
the morning, answering questions from the previous night, and posting 
answers that will be 'available when the student signs online that evening. 
Certainly, consistency in turnaround will be greater with the full -timer, 
leading to a more reliable completion rate. 

Support Communications 

As shown in Figure 14, the primary instructor had a great deal of 
telephone communication with the students. Phone use was heaviest during 
Phase 1; during Phase 2, we encouraged students to communicate online 
students calling with questions that could be answered online were told to 
post them on the computer. 

Early hotline calls to the instructor regarding hardware and software 
problems were quite heavy. Nearly 18% of all calls were placed in the first 
two weeks of the 31 week course. 

Over the duration of the course, students placed 388 calls to the 
instructor. This averages out to about 30 calls per student In 
mmoarison students initiated 426 online communications with the 
inXctor', or about 33 comments per student. Obviously, the telephone was 
a very important link to the instructor. 
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Figure 14. Calls to the instructor throughout the course 



Although the hotline was equipped with an answering machine that would 
take messages in the evenings and on weekends (with someone returning the 
call as soon as possible), students noted that the hotline was not available 
when they needed it most. This caused delays in their ability to complete 
materials—often they had to wait a day or more to get a solution to a 
software or hardware problem that was preventing them from working on the 
course. Further, since calls were usually returned during the day, students 
were away from their computers and could not work through the problem 
solution while on the phone, introducing further delays if the solution did 
not work Students appreciated having someone to help them during their 
hours of availability: "The hotline was generally available when you're not 
on the system because you're on the system late at night and on weekends. 
You guys have your regular office hours so if I have a problem at night I 
have to wait till the next day to go to work and call. Of course, it was 
'What's the problem' and I'd go home and check. I guess it was Bob I ended 
up working with a lot. He ended up staying late in the evening which was 
super. " 

It was also noted that more telephone contact would have been helpful 
between students. In our course, we did not encourage such communication 
for several reasons. First, we wanted to maximize use of the computer 
conference both so that we could better observe student interactions and 
because that was the medium of interest in the research. Second, we wanted 
to minimize telephone costs and were afraid that students would not restrict 
their conversations to those which were necessary. Hence, while we did not 
ban telephone use, we did not encourage it. Most students did not call one 
another until during the course culminating practical exercise. 

Students were of the opinion that telephone contact, especially early 
in the course, would have been helpful in building rapport: "More of a 
cohesion occurred when we got on the phone and discussed what was expected." 
Also, small points of coordination can be accomplished more efficiently on 
the phone than online, due to the inherent time delay of asynchronous 
communication: "I think that if you do this in the future you should make 
it clear to people that if they're going to coordinate little things using 
the phone works best." 

Students expressed anxiety due to not knowing classmates and not being 
able to get a feel for them quickly online: "I think calling and talking 
voice to voice first would help, personal interaction. Then when you go to 
the computer you can do more. I felt funny because I didn't know these 
guys." In interviews, however, students were unsure of whether a 
face-to-face meeting at the very beginning of the course would be 
beneficial. They were afraid that the experience would be too short and 
that they would not have time to develop personal relationships. Several 
students mentioned the value of getting to know one another online first 
then having a face-to-face meeting to cement personal bonds prior to 
beginning a course culminating practical exercise. 
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Support communications other than telephone are necessary, especially 
in cases where students must turn in assignments involving graphics which 
Ire difficult to draw on a computer screen. US mail is probably the most 
economical route, however, use of the mail involves delays of perhaps as 
long as a week. This has a negative impact on the rate of course 
completion: "I have completed the homework but will not be able to go on to 
the next topic until you get it in the mail and grade it and post it... Can 
we speed it up a bit?" 

Further, use of the mail may serve to shorten the time available to 
complete an activity, unless postmark date (rather than receipt date) is 
Taken as proof that an assignment was completed on time. This may not be 
feasible if the instructor needs to have the materials available for viewing 
asthe students participate in a related activity, such as a course 
culminating practical exercise briefing. 

We also tried allowing students to transmit assignments via telefax to 
allow them the maximum time to complete activities while still having the 
work available for instructor review in a timely fashion. However, no 
students used this method. It is likely that the students did not have fax 
machines readily available to them. However, as this technology becomes 
increasingly available it should also be^ -me more appropriate for use in 
SMART. 

Discussion 

In this section, an overall evaluation of SMART in terms of throughput, 
performance, and acceptability will be provided. This eva uatior i wil I be 
taken from the viewpoint of what the Army could expect in terms of these 
outcome measures if they were to implement a SMART course. In addition, 
Wessons learned from our experience will be noted. These should be helpful 
in improving future implementations. 

Before beginning the evaluation of SMART, however, it /^instructive to 
review how an Army implementation of a SMART course might look. For the 
purpose of illustration, we will show how the Eng neer Of f^er Advanced 
Course for Reserve Components (EOAC-RC) would be implemented n SMART . EOAC 
is being used as an example only; the concepts discussed should generalize 
to any RC course. 

What Would an Armv Tmnlementat inn of a SMART Course Look Like? 

The following illustrates how the three phase Officer Advanced Course 
for Reserve Components (OAC-RC) could be implemented for Engineer branch 
officers in SMART. Currently, EOAC-RC consists of: 

Phase I (the Reserve Component Company Command Course). A two 
week phase in residence at the Engineer School (USAES), it is an 
intensive technical, tactical, and training management oriented 




course designed to provide a newly assigned or prospective company 
commander with the skills necessary to train and lead the unit. 

Phase IIA (common leader training). This phase may be taken 
through a Reserve Forces (RF) school, by correspondence, or as a 
four week (two module) session in residence at the USAES. 

Phase I IB (branch specific training). Usually taken by 
correspondence, it may also be taken by attending a four week (two 
module) session in residence at the USAES. 

Phase III (RF school resident training). Individuals will be 
tracked separately during Phase III based on unit of assignment. 
Those assigned to combat engineer units will take Phase 1 1 1 A while 
those assigned to combat heavy battalions or construction support 
companies will attend Phase 1 1 1 B . Individual Ready Reservists may 
choose either track. USAES resident training can be substituted 
for either phase, with IIIA requiring four weeks (two modules) and 
1 1 IB two weeks (one module). 

SMART could be implemented at the course level such that EOAC-RC would 
be administered almost entirely through SMART. (Note - it is important to 
recognize that SMART can incorporate face-to-face instruction.) Figure 15 
shows a SMART implementation structure. 

(A) The Engineer School's Resident Company Command Course would be 
expanded by as much as two days to include face-to-face computer training to 
familiarize students with SMART methods and techniques prior to entering any 
computer-mediated phase. The familiarization would continue "remotely" via 
actual computer conferencing after the students' departure and before 
beginning another phase. 

(B) Each of the remaining phases would be delivered using SMART and be 
broken down into two three month blocks. Students could sign up for the 
phases in any order, but would, once in a phase, take the blocks in a 
particular sequence. Students could elect to take as much as one month off 
between blocks, but would need to schedule breaks carefully so as to 
complete the entire course within the prescribed limit (perhaps 30 months 
rather than the current 24). One recommended exception would be that no 
time-off be allowed between Phase I and the next phase to minimize the decay 
of computer skills taught during Phase I. 

(C) Topics within a block would be taken in sequence. Each topic 
might conUin both individual and group work, with group activities normally 
occurring in parallel (rather than serially) with individual activities. 
Major group activities such as the EOAC "capstone" exercises, which serve as 
the culmination of a block or phase, could be placed at the end of the 
section where they would logically fit. 
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(D) Elective engineer topics offered by the USAES could be made 
available either at the time of enrollment or following the main course if 
students require additional/refresher training and have access to a 
computer. 

In this implementation, a ceiling for class size could be determined 
from the number of students expected in the system at any given time. It 
may also be prudent to set a lower limit (e.g., 10 students) so that: (1) a 
"critical mass" is available for performing group activities and (2) each 
block can be taught in a cost effective manner. 

Now, let us see what outcomes (throughput, performance, acceptability 
and cost) the Army could expect if it were to adopt a similar SMART 
implementation. 

Throughput 

SMART classes will take longer for students to complete than will the 
equivalent resident course. In our study, a two week resident course took 
seven months to present via SMART. In an Army implementation, we would 
expect students to be able to cover this material in about five months. 
Remember that we had additional time added to our schedule in Phase 1, 
because students were working at their own pace. Given that students will 
have deadlines in an Army implementation, and that some subjects (such as 
the Allied Presentation and perhaps the Army Writing Program) would be 
offered as electives, five months would be a reasonable completion time. 

In our course, throughput, in terms of the percentage of students 
finishing, was inferior to that in resident programs. In resident school, 
95% or more of those who start the course finish it. In our course, only 
64% of the starters finished. This figure is probably the lower limit of 
what would be experienced in an Army implementation of SMART. Problems that 
plagued our course (such as software problems that were frustrating to 
novice computer users and the inaccurate estimate of the time commitment 
needed to complete the course) would not be present in a fully developed 
Army course. 

The resident school does not have a problem with non-starters 
students who receive slots in resident school attend in all but the most 
extreme circumstances. The SMART course did have quite a few non-starters 
(33% of those who signed up). Again, however, in an Army implementation it 
is unlikely that the problem would be so severe. First, delays in software 
development that required postponing the start date of our course would not 
be expected in an Army implementation -- as was stated previously, this 
accounted for a majority of the decisions to drop the course prior to the 
start date. Also, our students had other alternatives for taking the 
course, namely, correspondence. If the Army were to implement SMART, these 
other alternatives might not be available. 




Performance 

SMART training provides the same quality of training as that found in 
resident school. When performance of SMART students on the most appropriate 
available data was compared to the performance of resident school students, 
no significant differences were found. 

Unlike the comparison between performance of students in the pilot test 
with ACC students, this lack of a significant difference in test performance 
cannot be attributed to ceiling effects. Both SMART and Resident School 
test score data contained at least a 30 point range. 

Finally, SMART students were significantly more confident of their 
learning in their skills assessment. SMART students perceived a greater 
learning benefit than did resident students. 

Acceotabil ity 

In spite of the fact that students found the SMART course to be very 
demandinq, they had positive opinions of it as a learning experience. 
Virtua ly all completers preferred SMART to correspondence And, several of 
the completers compared SMART favorably with their resident experiences. 

Even those students who still had a preference for resident courses 
oraised the availability offered by SMART courses: "I like this system 
tetter than Correspondence courses. It is fun to talk with others and get 
in to the humor in uniform (borrowed this from Reader's Digest). I also 
like to go to Belvoir or even Ft. Wood to take classes with others. But it 
was hard for me to get away more than once in a few years to twc > summer 
camps or not go to camp with the unit so that I could go to class. This is 
a good compromise." 

Throughput, performance, and acceptability will be maximized only wi th 
careful attention to how the course is implemented. The following section 
documents the lessons we learned about course implementation. 

Lessons Learned 

We learned a great deal, both from the quantitative data collection and 
from observations of the impact of how we set course parameters The 
purpose of this section is to note these lessons for "corporation into 
future implementations of SMART. Specific guidelines on the how to s of 
implementation are given in other documents produced by this project. These 
documents, and their contents, are described subsequently. 

firmm interaction . Based on findings reported in the literature that 
peer interaction contributes to high morale and positive earn ; n 9 
experiences in remote training, it is reasonable to expect that courses 
which incorporate group learning activities in addition to individual 
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activities would result in higher throughput and acceptability than courses 
based only on individual learning activities. 

Although group activities have positive benefits in terms of pacing and 
enhancing student motivation, they reduce the students' ability to set their 
own schedule for task performance, reducing the SMART strength ot 
flexibility. Thus, a mix of individual and group activities should be 
pi anned. 

Further all group activities should be planned to occur in parallel 
with individual activities, rather than serially. In this way, students 
will be able to continue to progress through the course even if they must 
wait for asynchronous communications from group members. 

Pacing. Deadlines are indispensable for keeping students moving 
througlTTtourse. Deadlines must be set far enough apart to give students a 
reasonable amount of flexibility in the schedule they will set for 
accomplishing the objectives, but not so far apart as to encourage 
procrastination. 

Group activities also serve as a valuable pacing aid. Students will 
try to be caught up so as to participate fully in these activities and not 
let their group members down. These types of assignments should be 
interspersed throughout the course, at intervals which facilitate 
flexibility of scheduling for the individual student. 

Tvnes of course requirements . Both content- and process-oriented 
activities can be conducted successfully via SMART. However, if 
process-oriented activities come at the end of a content-oriented course, as 
was the case with our course culminating practical exercise, special 
attention is required to ensure that students have the proper expectations 
reqarding the activity. For the duration of our course, everything the 
students had been taught was "cookbook" and did not require integration on 
the students' part. Thus, a "cookbook norm" was established and the 
practical exercise, which required a great deal of integration, was totally 
out of character with what the students had been doing previously. Thus, a 
clear statement of the objectives of the exercises, as well as a description 
of how students should proceed in order to meet those objectives, are 
needed. Further, providing students a break down of the grading of each 
aspect of the exercise may be effective in helping them to place proper 
emphasis on the various exercise activities. 

Further, careful consideration must be given as to whether or not 
students have learned everything needed to complete the exercise in earlier 
portions of the course. Specifically, consideration must be given to 
whether students have sufficient background in the process aspects of the 
exercise. For example, in our course students said that they did not know 
what was expected of them during the briefing portion of the exercise since 
no other briefings had been given previously in the course. Hence, a dry 
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run of the briefing would have helped prepare them with respect to the 
procedures expected. 

Finally, student expectations pro^Jt Wjlp "^f 

of the process-oriented .activities, if 1 l t 2ients had Kd interactions 
previously experienced in the course ^students naa unp ieasantly 

student schedules. 

that eight hours included course content as wel « J^ 1 ^ time required 
such as uploading and downloading. Careful esnmau lun u 
for both 'technical and administrative taste tu ^^J 1 ^.^^^ completers, 
throughput, both in terms o ra e o f compl ti oj^d numbers J ^ 
As has been P^viously state . Jhe extr a time comm ^ ^ 

our course was one reason for attrition ciueu vy 



course. 



n a « si ? P Short content-oriented activities can be accomplished by 

unreasonable tor tne exercise, *.uu QVQrr ^ cp 
work on each content-oriented problem contained in the exercise. 

Further in every class there will be a certain number of students who 
runner, in eve.j in nur rla<;<; one or two team 

do not participate in group activities. In our class one o 

STS ^tW^™ "ac fSMKi olrTudenU ^s available for 
each activity. 

Variety, of media . With the exception of TH and FM readings all media 
used i n the course had both detractors and "P^^'^^^^S^ prefer 
preferences among the media were expressed - Hoover, udents d *p & 

"omputeT »S»E Slffl^S over those which 

required instructor grading. 

Students also liked group activities, particularly those that occurred 
in the synchronous mode. 
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Learning skills and strategies . The primary lesson here is that 
students become very reliant on the cognitive organizers provided for them 
in the course. When we shifted the type of organizer (e.g., from 
computer-based to paper-based), students became confused they did not use 
the paper-based organizers provided for the practical exercise since they 
had become so accustomed to having tasks listed for them on the computer. 
Thus, organizers provided to the students must be consistent throughout the 
course and must also be very specific with respect to what is required of 
students. 

Also, because the primary source of student anxiety revolves around 
unfamil iarity with the computer and lack of computer skills, courses should 
be designed to build computer skills incrementally. 

Feedback . Students preferred to have immediate feedback on learning 
activities. The need to wait 24 hours for feedback on some activities was 
very frustrating. However, this problem could be ameliorated by allowing 
students to continue on to other activities while awaiting feedback. Also, 
the availability of support communications to quickly solve problems is 
critical. 

Given the asynchronous nature of computer communications, and the fact 
that RC students are likely to only work on the course in the evenings or on 
weekends, it is probably not realistic to expect turnaround to occur in less 
than 24 hours. Setting the instructor's work schedule such that most online 
work is done in the mornings is adaptive in meeting a 24 hour turnaround 
goal replies to the previous night's questions and comments can be 
waiting for students when they log on that evening. However, some 
instructor time must also be made available during the hours when students 
are most likely to be online. Evening and weekend online "office hours" may 
help meet this need. 

Motivational strategies . Of the motivational techniques we tried, the 
most successful were having a penalty for turning assignments in late and 
providing feedback about team participation. We recommend that these 
strategies be employed in other SMART implementations. Penalties for late 
assignments should be in conformance with the policy of the appropriate 
resident school . 

Instructor characteristics . As was previously noted, we had very mixed 
success with our part-time instructors. However, our experience does not 
provide enough data to recommend for or against using part-timers. Rather, 
more research is needed into this issue. If the Army were to use 
part-timers, they would have far more leverage for ensuring that the 
instructors met their commitment than was available to us. In these more 
constrained circumstances, the use of part-timers might be more successful. 

Support communications . The availability of other means to communicate 
with the instructor (i.e., telephone) is critical. Telephone communication 
provides a faster way to solve student problems than online communication, 
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and may assist students in meeting tight deadlines. Even more important is 
that fact that students experiencing computer difficulties may be unable to 
communicate online. These students need an alternative means of reaching 
the instructor for help. 

Students also expressed concern over not knowing their classmates. 
Getting to know one another online takes a considerable amount of time. 
Telephone contact in the beginning of the course can speed this process. 
Opinions on the value of face-to-face meetings at the beginning of the 
course were mixed. Unless such face-to-face meetings are long enough for 
students to really get to know one another, the expense is probably not 
warranted. 

Like online interactions with the instructor, support communications 
must be available during the times when students most need them (i.e., 
evenings and weekends). This is particularly critical for students having 
computer problems, as they need to be able to call for help when they have 
ready access to their machine so they can be "walked through" potential 
solutions. 

It is also helpful to provide means of transmitting graphic data that 
cannot be typed on a computer screen. US mail and fax service can meet this 
need. 

As was mentioned previously, these lessons learned are provided to 
sensitize readers to issues that they need to consider to implement a SMART 
course. Specific guidance on the "how to's" of implementation is provided 
in two other documents generated by this project. These are described 
below. 

Guidance Documents 

Distributed Training for the Reserve Component: I nstructor Handbook for 
Computer Conferencing (Harbour, Daveline, Schurman, Richards, Hahn, and 
Wells, 1990). This document is a job aid for instructors, trainers, and 
other' key personnel working in a SMART environment. It provides general 
information about the SMART environment, the role of the SMART instructor, 
characteristics of adult learners, principles of learning, and information 
on the identification of high risk students. In addition, "how to" 
information on establishing class norms, group learning techniques, 
individual instructional techniques, student motivation, online etiquette, 
and course preparation is given. Finally, troubleshooting guidance on 
hardware and software problems, the SMART environment itself, group and 
individual instruction, normal class operations, and common student problems 
are discussed. 

Distributed Training for the Reserve Component: Cou rse Conversion and 
Implementation Guidelines (Hahn, Harbour, Schurman, Daveline, and Wells, 
1990). This document is intended to provide principles and context for 
course developers and managers for developing and managing SMART courses in 
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accordance with the SAT (Systems Approach to Training) process. It provides 
recommendations for the analysis, design, and development of SMART courses 
and notes additions to or special considerations in the SAT process for 
SMART course development. In addition, recommendations for implementation 
of SMART courses are given with respect to: (1) when to use SMART courses; 

(2) who should use SMART courses -- instructor and student requirements; and 

(3) how SMART courses should be implemented with respect to technological, 
pedagogical, and administrative considerations. 

Cost Analysis for SMART Implementation 

In previous sections of this report, it was demonstrated that SMART is 
an effective training method with respect to throughput, performance, and 
acceptability. However, effectiveness alone will not form the basis for a 
decision as to whether or not to implement a SMART program. Cost, too, must 
be considered. This section reports on two cost analyses: (1) what costs 
would be expected if the Army were to implement SMART; and (2) the 
cost-effectiveness of SMART compared to resident training. 

Implementation Costs 

In determining costs for an Army implementation effort, costs were 
extrapolated from our actual experience and were based on the assumption 
that thp phases of the implementation model described in the "Discussion 
section are equivalent in terms of material presented and duration to Module 
6, EOAC, as delivered via SMART . Further, it was assumed that each phase 
outlined in the model would have a full-time instructor plus other 
instructional, administrative and logistical support and that 50 students 
would be taking each phase at any given time. 

Cost analyses were divided into sections for course conversion and 
course execution. Costs associated with the conversion of resident 
materials for SMART courses were derived from a report prepared by the 
TRADOC Management Engineer Activity titled "Estimated Time Values [ETVs] for 
Training Developments" (1984). These costs are expressed in professional 
staff hours Course execution costs were based generally on the cost 
categories defined by Knapp and Orlansky (1983). These costs are expressed 
in dollars The "course" is assumed to be equivalent to one module ot tUAl 
(two weeks of instruction totalling 66 classroom hours). It is also assumed 
that a software package would be available to implement SMART so software 
development costs are not included in any of our projections. 

Table 9 lists the estimate of the number of hours it would take a 
branch school to duplicate the course conversion accomplished by the SMART 
project staff. The conversion represents a revision of existing materials 
rather than full scale development. For purposes of comparison, we have 
also used the TRADOC document on ETVs to estimate the number of hours 
required to develop the same program of instruction from scratch for 
presentation in residence using traditional instruction methods. All the 
individual categories and subcategories for full scale development are not 
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Time (Hours) 



Table 9 

Fstimated Amount of Tim e for SMART Conversion 
Category 

Course Requirements Analysis 
Conduct Initial Review 
Assemble Draft Training Program 

Package 
Coordinate Draft Training 

Program Approval 
Provide Draft Training Program 

Implementation Assistance 



Course Design 
Select Instructional 

Setting 
Identify Learning Objective 

Relationship 
Group Learning Objectives 
Prepare Practical Exercise 

Tests 

Course Development 
Develop Oraft Course Management 
Plan 

Revise Draft Course Management 
Plan 

Oevelop New Lesson Plan 
Develop New Student Aids 
Develop New Instructor Guide 
Validation - Presents Lesson 
Validation - Revises Lesson 
Proofreads Final Product 

CBI/Slide Conversion 

Video Tape Production * 

Total 



111 
180 
42 
102 

46 

20 
15 

82 

101 

19 
459 
576 

81 
475 
325 
553 

812 

251 

4250 



% of Conversion Effort 
10% 



4% 



61% 



19% 
6% 



Resident Course Development Total 



11761 



* Includes the hours to prepare and supervise the contract for a CBI P^ct 
plus the hours spent by the ACC Project contractor to actually perform the low 
level CBI conversion reguired by the ACC Project. 
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shown, but have been combined to produce the single figure shown at the 
bottom of the table. 

Projected course execution costs are shown in Table 10 with separate 
columns for one time and recurring costs. Each column was further 
subdivided into contractor and in-house sections. Actual hours the 
contractor spent working on the project were adjusted by cost category to 
eliminate research peculiar activities and to expand the projection to cover 
two modules and the increased student load implementation would incur. The 
hours were then multiplied by the contractor's average "loaded" hourly rate 
in dollars and the rate calculated for an in-house effort from standard pay 
tables and data on operating costs provided by TRADOC (ATRM-159 Report). 
The production, equipment, and shipping categories are the same for 
contractor and in-house because we have assumed that the branch school would 
absorb essentially the same costs experienced by the contractor. 

The execution cost categories have been defined as follows: (1) 
Production -- reproduction of materials and preparations for shipping; (2) 
Equipment -- computer hardware and software; (3) Training -- training 
instructors, both face-to-face and online; (4) Shipping -- transporting 
computers and materials to students; and (5) Operating and Support -- 
instructor salaries, computer maintenance and communications, support 
communications, course management, and test and evaluation. 

Cost Effectiveness 

Table 11 shows course execution costs in dollars per student and 
dollars per student per hour of instruction. Again, an in-house / 
contractor range is provided for the SMART implementation. SMART costs were 
based on a class size of 50 and one time costs were amortized over five 
classes because it can be expected that course materials and equipment would 
be reused. The resident costs shown are based on the ATRM-159 Report. 

Table 10 



TRADOC SMART Execution Costs 



Execution Cost 


One Time $K ** 
(Contractor / In-House) 


Recurrinq $K 
(Contractor / In-House) 


Production 
Equipment 
Training 
Shipping 
Operating and 


49.3 / 49.3 
46.7 / 23.8 

Support 


16.8 / 16.8 

0.6 / 0.3 
8.0 / 8.0 
189.0 / 96.1 


Totals 


96.0 / 73.1 


214.4 / 121.2 



** It has been assumed that new training and equipment will be required 



after five course iterations. 
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Table 11 

Relative Costs of RC Training Options 



Executi on 

S/Student $/Student/Hour 

SMART 4,672 (contracted) / 2,716 (in-hou,e) 71 / 41 

Resident 5,793 



Relative costs alone, however, are not adequate to determine the 
viability of SMART for RC training. Both cost and effectiveness measures 
(throughput and quality) were directly compared. 

As was shown previously SMART courses are comparable to resident 
courses with respect to quality (performance and acceptability) Hence, 
since execution is less expensive for SMART than for resident school, it 
louU compare favorably on'cost-ef fecti veness (CE) ratios based on these 
measures . 

The critical CE ratio, however, is that concerned with throughput, 
since it is in this area that resident school surpasses SMART Previously, 
11 "gave conservative estimates of SMART throughput at 64% and c >f resi dent 
throughput at 95%. Given a starting class size of 50, the CE ratios can be 
developed using the following formula: 

CE = (Class Size Percent Throughput)/Cost x 100% 

This ratio reflects the execution cost per student for students who actually 
become trained. 

For SMART, CE (contracted) = 0.68 and CE (in-house) - 1.18. ™ e 
resident CE - 0.82 see Table 12). The SMART CE ranges demonstrates that 
SMART is at least comparable to residence in cost effectiveness at even 64/, 
throughput. However, as has been previously stated, we feel that SMART has 
the Potential to achieve better than 64% throughput. If this is the case 
SMART will c early exceed resident CE. Figure 16 shows that even contractor 
ev" loM and conducted SMART training at 64% throughput -o-s ess cos ly 
than resident training after four course iterations. In this illustration, 
one time front-end conversion/development costs have been added to one time 
and recurring execution costs at the first iterations for SMART training 
One Ume execution costs are repeated after the fifth iteration because it 
has been assumed that there will be a need both to train new instructors due 
to personnel turnover and to replace worn-out equipment. Dollar amounts for 
conversion/development were determined by multip ying the hourly rates 
previously discussed by ETV hours. An additional amount ($34 IK), equal for 
hl!h "SnntrJ or" and "in-house" efforts, was added to cover the cost of 
^«tS B baS°1ns^t1on (CBI) development. In Table 9, it was assumed 

69 



ERIC 



61 



Table 12 

Cost Effectiveness Ratios 



Throughput: SMART 64% 
Resident 95% 

50 x .64 

Cost Effectiveness Ratios: CE (SMART in-house) = x 100 = 1.18 

2716 

50 x .64 

CE (SMART contracted) = x 100 = .68 

4672 

50 x .95 

CE (Resident) = x 100 = .82 

5793 



j ooo jn 

tsooj ^^^^^ r^-^*^^*^ * 



t 2 3 4 % 6 7 8 9 -0 

ITERATIONS 

SMART-in House SMART-Contracted Resident 
— # — — Q — — B — 

Figure 16. Implementation costs 
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that CBI development would be contracted out rather than conducted 
in-house. 

Conclusions and Recommendations 

In this report, it has been demonstrated that SMART courses are 
effective, as compared to resident programs, with respect to performance and 
acceptability. While SMART courses have not yet achieved the throughput 
seen in resident programs, their throughput does exceed that ot 
correspondence courses, while providing the same flexibility afforded by 
correspondence. Indeed, SMART courses suit student's needs for flexibility 
of scheduling and are generally more available to a wider range of RC 
students than are resident programs. 

Further, the cost effectiveness of contractor developed SMART programs 
is similar to that of resident school, in spite of the lower throughput. 

SMART is very effective in meeting the needs of the RC because it can: 

-- overcome geographical dispersion and low density MOS training 
retirements by brinqing students together remotely; 

r6qU stretch training resources (instructors) by making an instructor 

in one location available to students in many locations; 

-- accommodate civilian and personal commitments, due to its 

asynchronous nature; and 

provide a means for maintaining Army contacts online. 

However, SMART courses will not be successful unless certain 
implementation requirements are met. These include: 

-- providing each student with a suitably equipped computer, 
Dreferably a portable rather than a desk top model; 

providing wel 1 -supported, wel 1 -documented software to support 
communications, conferencing, and special appl Rations; inrlluHna 

-- giving special attention to logistical coordination, including 
access to telephone lines and adequate power supplies; 

- setting deadlines for completion of activity, and proving both 

inCent l V ^g P ^^ techniques 
such as Performance feedback;^ ^ ^ Qf Qr proce$s 

emphasis) and explicitly stating requirements; 

-- scheduling the course to accommodate civilian jobs, family 
commitments, and other military duties; 4 ^ 4u4 H.. a i 

-- using a variety of media, including both group and individual 
learning activities, for the presentation of instruction; and 

- providing support communications (i.e., a telephone hotlme) wfnch 
are available during hours when students are most likely to be working on 
the course. 
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Further details on implementation requirements can be found in 
"Distributed Training for the Reserve Component: Course Conversion and 
Implementation Guidelines" (Hahn, Harbour, Schurman, Daveline, and Wells, 
1990). 

Questions still remain regarding the viability of a course level 
implementation of SMART . Specifically, more information is needed on: 

Staffing requirements: what mix of full- and part-time 
instructional and administrative support is needed to conduct a course level 
implementation of SMART? 

Software interface: what are the optimal software requirements 
with respect to communication, delivery of instruction, and course 

administration? and 

Class size: how many students can one instructor handle in a 
classroom setting and in a team room setting? 

Based on the results found in the implementation test, SMART appears to 
be able to provide acceptable throughput, performance, and availability at a 
reasonable cost. The use of SMART for Army training should be further 
pursued. 
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A SAMPLE SMART SESSION 

SMART presents a very different milieu than the face-to-face 
classroom. The following descriptions are based on features found in the 
software implementation created for the EOAC course. They are intended only 
as an example of what SMART software features might look like. To 
illustrate this, Figure Al shows what the SMART student sees and does during 
the computer session that takes place at 2030 hours on Thursday. 

(A) The first thing the student sees after turning on the computer is 
an analog of an electronic school. This school contains "rooms" or places 
where various activities occur -- feedback on tests is given in the office; 
small group activities take place in the team room; class discussions take 
place in the classroom; general chit-chat takes place in the break room; 
self-contained lessons, such as CAls and storyboards, are taken in the 
learning center; and homework assignments and tests are done in the writing 
center. The school metaphor serves as an aid to understanding the learning 
environment for the student who is moving from the known world of the 
face-to-face classroom to the unknown world of SMART. 

(B) The task list serves as an organizer, whereby the student selects 
activities to perform. In addition to selecting self-contained course 
requirements, such as performing a CAI, the student may elect to send or 
receive information to/from other students and/or the instructor. This is 
done by "uploading" or "downloading" to/from the host computer. The task 
list also serves a course management function as it keeps track of what 
lesson activities the student still needs to complete. 

(C) A typical student would probably first elect to download 
information from the host computer. By downloading, answers to any 
questions that were posed to the instructor, messages from classmates, and 
feedback on any graded activities completed would be received. Only that 
information sent to the host computer since the student last downloaded is 
provided. Information is organized hierarchically by net, item, response, 
and date. Before continuing through our overview of the SMART session, an 
explanation of these concepts is in order. 

A net is the computer equivalent of a room in the electronic school 
where activities take place. Information from similar types of activities 
are stored together in a net. For example, in Figure Al (C), the net is the 
CLASSROOM. This is where discussions about technical topics take place. As 
can be seen from the figure, students are asking questions about course 
content . 

An item is a statement or discussion about a single subject or 
thought. In Figure Al (C), you will note designators such as 14:4 and 
17-5 The number to the left of the colon is an item number which keys the 
student to the item being discussed. In the example, Item 14 is a technical 
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Break Room 


Classroom 


Team Room 


Writing 
Center 








Team 
Room 




Task List 




Learning 
Center 






Team 
Room 


Office 



TASK LIST 

1. Do reading on airfields 

2. Do CAI on airfields 

3. Review roads and airfields lessons 

4. Take roads and airfields exam 



STATUS 
DONE 
NEED 
NEED 
NEED 



A. Upload to hose 

B. Download from host 



Select menu option 



Downloading from CLASSROOM: 
Feb21/89 23:08 

14:4) John Jones: Help on activity 8 
When I fill out the table in step (b-2), I get a No 
Go for the No. 10 sieve and Gos for the rest. The 
tables values are 80, 15, 25, and 5. The sample 
values are 80, 10, 15, and 4. Where am I going 
wrong? I'll check back later tonight for some 
advice and will drive on in the meantime. 

Feb. 21/89 23:45 

17:5) Mike Smith: I need help too 
Help! I do not have a clue as to what is going on 
in the thickness design of each layer in the 
building of a road or airfield. It is not clicking 
as to where the thickness is coming from or how to 
break down each layer and know how thick it should 
be. I understand the first part of the problems, 
but not the last 2 or 3 steps. Can anyone help??? 

Feb22/89 00:08 

17:6) Joan Black: Here is a little help John 
John, I can't help you a lot but if you use half of 
the months at 31 days and half at 30 you will come 
up with the correct answer. 



igure Al . Sample SMART session 
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discussion on Road Design, while Item 17 is a technical discussion on 
Airfield Design. 

In the designators, the number to the right of the colon is a response 
number A response is a comment about an item or another response. 
Responses are displayed in the order in which they are received, as 
indicated by the date and time stamps shown in Figure Al (C). 

The excerpts shown in Figure Al (C) are transcripts from a real class, 
with only the names changed. They represent a portion of the 
student-to-student dialog taking place in about a one-hour period on a given 
evening. Several features of the transcript are noteworthy: 

The time stamps -- as was staled previously, SMART students do 
most of their course work during evening and weekend hours; the 
time stamps reflect Eastern time; "John Jones," a student who 
lives in Maryland, was working on the course at 11 p.m.; 

Organization -- because students are working asynchronously, even 
though responses 17:5 and 17:6 follow one another chronologically 
they do not follow logically; response 17:6 obviously is a 
response to a comment posted at some time previous to 17:5; 
further, not all students are working on the same course topics, 
as evidenced by activity in two items which do not refer to the 
same content areas; the result of this is that students must 
devote time and effort to organizing downloaded information into a 
logical (i.e., by item) sequence, usually by printing the 
information and binding it in a notebook; and 

Peer learning -- these items are being used by the students to ask 
and answer technical questions; while the instructor would have 
the ultimate responsibility of answering technical questions, 
other resources are available to the student with a question, as 
evidenced by "Joan Black's" response; unlike the face-to-face 
environment, where it is usual for the instructor to speak and the 
student to listen, computer conferencing affords equal air time to 
all and students are free to respond to one another in any public 
discussion. 

Returning to the discussion of how a SMART session appears to the 
student, let us examine a snapshot of a the student's remaining tasks, as 
shown in Figure A2. 

(A) The SMART software prompts the student for comments on items and 
responses that were just read. The software transforms the student's entry 
into language understandable by the conferencing system, so the student doe; 
not have to learn the peculiarities of a particular conferencing system. 
Finally, the SMART software returns the student to the tusk list when 
downloading and responding are completed. 
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Respond to item? Y/N 
Item number? 14 
Enter your response: 



John, what CBR are you using for a table 
value? Remember to downgrade the CBR 
and recheck it. 



MORF/ RFTtIRN TO TASK LIST 



DESIGN OF AIRFIELDS 

This set of lessons will introduce you to the design 
of airfields. Please proceed through the lessons in 
alphabetical order. 



A. Introduction 

B. Design Steps 



Select menu option 



STEPS IN DETERMINING RUNWAY LENGTH 130 

The TGR in Table 3 was determined at 59 degrees F, 
at an altitude of 0 feet, and with an effective 
gradient of less than 2%. If any of these three 
factors are different for the airfield being 
designed, correction factors will have to be 
developed. 

Press Enter for more 



STEPS IN DETERMINING RUNWAY LENGTH 135 

Which one of the following is NOT a correction 
factor that may have to be applied to the TGR? 

A. temperature C. effective gradient 

B. altitude 0- wind strength 

Enter the selection of your choice: D Correct! 



Figure A2. Continuation of sample SMART session 



ERIC 



71 



(B) The next task selected by the student is performing a CAI lesson. 
The SMART software automatically finds and displays the correct lesson for 
the student. As the student moves through the lesson, information is 
presented and questions are asked. When the student answers a question, 
immediate feedback is given as to whether or not the answer was correct. 

Serving the SMART function as a course management system, the SMART 
software stores the student's answers and automatically sends them to the 
instructor when the student next selects the upload function from the task 
list In fact selecting the upload function causes all newly created data, 
such' as the responses shown in Figure A2 (A) and answers to CAIs and 
quizzes, to be automatically transmitted to the host computer in the format 
required by the conferencing system. Typically, the student would upload 
after completing all other computer-mediated activities during a particular 
session. 
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APPENDIX B 



SMART SOFTWARE IMPLEMENTATION 

Because Phases 1 and 2 differed with respect to the availability of 
group interaction, slightly different versions of the electronic classroom 
were presented via SMART for the two phases. Recall from the previous 
discussion, that the functions of SMART are to provide a communication 
system, a combination of delivery media, and a course management system. 

The course management function was fully supported in both phases. In 
both phases, a task list was used to guide student access to the learning 
activities. This task list provided a display of personal progress through 
the course for each student. An office function was also available in both 
phases; this provided a place where the student could obtain feedback on 
grades and course progress. 

To a lesser extent, a variety of media were also supported in both 
ph? ps. Both phases fully supported paper-based and locally delivered 
electronic media, such as CAIs and storyboards, through the learning 
cer er. Phase 1 was more limited than Phase 2, however, in that no 
synchronous or asynchronous group activities were supported. 

This point ties in to the main area in which SMART functions were not 
fully supported in Phase 1 -- communication. In both phases, homework and 
other written assignments to be sent privately to the instructor were 
pc- -rmed in the writing center. In Phase 1, however, the many-to-many 
co' nication supported by ACC was curtailed such that students could only 
huiu private discussions with the instructor or another student. No group 
communication was possible. 

The following paragraphs detail the features of SMART that were unique 
t< jch phase. 

P'i ^ se 1 

Figure Bl (A) shows the analog of the electronic school displayed for 
il.f students during Phase 1. In this school, content-related private 
H'scussions between a student and the instructor took place in the 

<:room. One-on-one conversations on any topic could take place between 
ents or between a student and the instructor in the break room. 

Students entered the school via the task list, shown as Figure Bl (B). 
that all activities on the task list are individual content-oriented 
(ties At the outset of the phase, students were not given deadlines 
iopic completion. Rather, they were told to work at least the minimum 
•;q< < hours per week and to complete the phase as rapidly as they could. 

* 2 

n Phase 2, many-to-many communications were supported via the INTERACT 
. u re shown in Figure B2 (A). This feature allowed for group 
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Break 
Room 


CI assroom 


Learning 
Center 


Writing 
Center 


Office 


Task List 





TASK LIST 

Rear Operations 

1. Read paper-based lesson 

2. Take quiz 

3. Read & outline stor,ooard 

4. Ream FM chapters 

5. Take quiz 

6. Review for exam 

7. Take exam 

A. Upload to host 

B. Download from host 



STATUS 

DONE 
DONE 
NEED 
NEED 
NEED 
NEED 
NEED 

Select menu option 



Figure Bl. Phase 1 electronic school 

communication in the break room (on any topic), class room (on 
content-related subjects), and in the team rooms (on group exercises) 
Interaction in the team rooms was semi-private in that it was limited to 
team members and the instructors, and could not be viewed by class members 
who were not part of the team. 

Different forms of private communication were also supported. 
One-on-one discussions could be held between students in the mail room, a 
student and the instructor in the instructor help room, and a student and 
the team leader in the team leader help room. 

As shown in Figure B2 (B), the task list for Phase 2 directed students 

to both individual and group content-oriented activities. At the outset of 

this phase, students were given due-dates for each topic, based on the eight 
hour per week time requirement. 

Phase 3 

In Phase 3, all features present in Phase 2 were available to the 
students However, the learning center was only used by students needing to 
review previously taught content -- no new instruction was presented in the 
learning center. Further, the task list was not presented electronically. 
Rather, to avoid costly software changes, students were sent a paper 
syllabus which detailed the tasks to be performed. The information provided 
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TASK LIST 

Flexible Pavements 

1. Read & outline storyboard 

2. Take quiz 

3. Perform paper-based exercise 

4. Take quiz 

5. Solve a team exercise 

6. Review for exam 

7. Take exam 

A. Upload to host 

B. Download from host 
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DONE 
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NEED 
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Figure B2. Phase 2 electronic school 



on the syllabus was the same as that provided electronically, except that 
students had to manually chart their progress through the activities. 
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